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The  Navy’s  individual  training  and 
education  system  is  grounded  in  a 
proven  process.  It  has  a  long  history 
of  providing  highly  qualified  and  motivated 
Sailors  to  the  fleet.  But  like  almost  every 
other  system  today  and,  particularly,  any 
training/education  system,  rapid  change  is 
affecting  almost  every  component  of  the 
process.  To  prepare  Sailors  for  the  21st  cen¬ 
tury,  the  individual  training  system  had  to  be 
reengineered.  This  was  accomplished 
through  quantifiable  methods  and  prudent 
risk  taking  to  enable  us  to  continue  to  deliver 
a  highly  qualified  technician  and  motivated 
Sailor  to  the  fleet. 

The  demographics  of  incoming  students 
from  today’s  environment  yield  more  diver¬ 
sity  in  factors  ranging  from  cultural  back¬ 
grounds,  exposure  to  technology,  gender, 
team  working  skills  and  a  host  of  other  soci¬ 
etal  factors.  The  rapid  rate  of  technological 
advancement  has  greatly  increased  the  sheer 
volume  of  “must  know”  technical  knowl¬ 
edge  essential  to  mission  accomphshment. 
The  increasing  use  of  cormnercial  off-the- 
shelf  equipment  has  significantly  reduced 
the  time  available  to  design  and  implement 
supporting  training  and  education.  The 
information  age  and  Network  Centric  War¬ 
fare  have  brought  a  requirement  for  increas¬ 
ing  skill  in  information  management  and 
more  complex  and  quicker  decision  making. 
All  of  the  above,  coupled  with  the  increasing 
variety  and  number  of  missions  and  the 
increased  emphasis  on  joint  and  multination¬ 
al  warfare,  also  portend  a  need  for  an 
increase  in  leadership  skills. 

The  driving  factors  hsted  above  as  well 
as  the  current  fiscal  environment  led  the 
Chief  of  Naval  Education  and  Training 
(CNET)  to  reach  the  following  conclusions. 
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Operational  Tempo  (OPTEMPO)  will 
remain  the  same  or  increase,  the  rate  of 
change  will  continue  to  accelerate  and  the 
Navy  cannot  afford  —  in  terms  of  resources, 
time  and  fleet  support  —  the  current  individ¬ 
ual  training  system.  The  current  system 
does  not  optimally  support  the  accession 
reality  that  45%  of  recruits  enter  the  Navy  in 
the  months  of  June  through  September.  Fac¬ 
tors  both  within  and  outside  the  control  of 
training  currently  result  in  executing  more 
than  the  training  and  education  command’s 
share  of  individual  account  resources.  The 
Navy’s  current  individual  training  system,  as 
currently  configured,  is  not  capable  of  effec¬ 
tively  and  efficiently  preparing  the  Sailors 
we  need  for  the  21st  century. 

The  question  confronting  us  is  how  do 
we  restore  productivity  to  the  individual 
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training/education  system  while  reducing  the 
amount  of  resources  needed  to  operate  the 
system  and  maintain  the  number  of  high 
quality  graduates  the  fleet  needs.  During  the 
analysis  on  factors  influencing  the  system, 
the  CNET  staff  reviewed  available  data  from 
numerous  sources  and  focused  their  efforts 
on  advances  in  training  technology  and 
methodology.  As  a  result,  the  Navy  is  cur¬ 
rently  experiencing  a  measurable  level  of 
success  with  initiatives  such  as  video-tele¬ 
training  and  computer-based  training.  New 
information  from  academia,  industry  and 
other  government  agencies  support  an 
assumption  that  a  systematic  approach  to 
applying  these  new  and  evolving  learning 
technologies  and  methods  will  have  a  major 
positive  impact  on  the  productivity  and  suc¬ 
cess  of  the  individual  training/education  sys¬ 
tem. 

To  begin  this  systematic  approach, 
CNET  developed  a  course  analysis  model 
that  established  criteria  for  evaluating  and 
categorizing  courses  based  on  their  technol¬ 
ogy  reengineering  potential.  For  example, 
courses  with  student  throughput  of  100  stu¬ 
dents  or  less,  that  were  90%  or  more  lecture 
in  content  and  less  than  three  days  in  length 
became  prime  candidates  for  video-teletrain¬ 
ing  delivery.  In  establishing  criteria  for  elec¬ 
tronic  classrooms,  courses  with  student 
throughput  greater  than  250,  with  a  mini¬ 
mum  of  60%  lecture  and  a  course  length  of 
8  weeks  or  longer,  provided  the  fastest  and 
greatest  return  on  investment.  Academic 
attrition  factors  and  setback  rates  were  then 
applied  to  establish  criteria  for  learning 
resource  centers,  focusing  on  courses  with 
attrition  greater  than  6%,  setbacks  greater 
then  10%  and  throughput  greater  than  200. 

(See  CHANGE,  p.  26) 
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Developing  the  Junior  Analyst  for  1998/99  —  End  of  Third  Quarter 


Three  months 
have  quickly 
passed  since 
the  last  article.  Since 
my  “halftime”  report, 
“your”  Society  has 
been  busy  preparing 
for  the  two  special 
meetings  before  our 
big  finale  —  the  67th 
MORSS  at  West 
Point  22-24  June 
1999.  We  are  happy  to  see  the  daily  inter¬ 
action  between  the  Senior  and  Junior  ana¬ 
lyst  going  on  within  the  Society  and  the 
Operations  Research  community  in  gener¬ 
al.  I  would  also  like  to  welcome  our 
newest  Director,  CAPT  Bob  Eberth, 
USNR.  Although  we  have  three  quarters 
of  the  game  behind  us,  the  fourth  quarter  is 
a  big  push  with  another  mini-symposium 
in  March,  the  Education  Colloquium  in 
April  and  final  preparation  for  the  67th 
MORSS  in  June. 

Also  keep  in  mind,  that  after  this  game 
ends  at  the  67th  MORSS,  we  must  start  to 
prepare  for  the  next  game,  which  is  fi’om 
the  67th  to  the  68th  MORSS.  President 
Elect  Dr.  Bob  Sheldon  is  presently  guiding 
this  effort.  The  Society  is  taking  a  strategic 
look  at  our  Special  Meeting  topics  for  the 
next  two  years  to  ensure  a  logical  flow  and 
MORS  is  meeting  the  needs  of  the  DoD 
analytical  community.  I  will  expound  on 
these  special  meeting  topics  in  my  next 
article,  after  I  have  received  concurrence 
from  our  six  DoD  Sponsors.  You  will  also 
hear  about  these  meetings  during  the  ple¬ 
nary  session  at  the  67th. 

Secretary 

Dr.  Tom  Allen,  Secretary  (Institute  for 
Defense  Analyses),  has  been  at  the  helm 
for  the  past  year  guiding  both  the  Member¬ 
ship  and  Electronic  Media  Committees, 
The  Membership  Committee  has  been  ably 
headed  by  Dr.  Dean  Hartley  (Data  Sys¬ 
tems  R&D  Program).  One  of  the  many 
important  functions  of  the  membership 
committee  is  to  nominate  the  Fellows  of 
our  Society.  This  year  the  Board  of  Direc¬ 


tors  elected  three  new  Fellows  of  the  Soci¬ 
ety  during  the  December  board  meeting. 
We  would  like  to  sincerely  congratulate 
Mr.  Brian  McEnany,  PP,  (SAIC);  Dr. 
Stuart  Starr  (MITRE),  and  Dr,  Harry 
Thie  (RAND).  These  three  “gentlemen” 
will  be  inducted  during  the  opening  ple¬ 
nary  session  at  the  67th  MORSS  in  June  at 
West  Point. 

Hopefully,  you’ve  had  an  opportunity 
to  visit  our  web  site  at  www.mors.org. 
This  web  site  has  made  great  strides  the 
past  few  years,  due  largely  to  the  efforts  of 
Ms.  Corrina  Ross  of  the  MORS  staff  and 
our  Electronic  Media  Chair  Mr.  Glen 
Johnson  (US  Arms  Control  and  Disarma¬ 
ment  Agency). 

Finance  and  Management 

Many  thanks  to  the  entire  membership, 
the  management  committee,  and  the 
MORS  staff  for  a  great  year  financially  in 
1998.  See  the  VP(FM)’s  article  by  Ms. 
Sue  Iwanski,  (Systems  Planning  and 
Analysis)  in  the  June  issue  for  details  on 
the  management  committee  composition 
and  their  accomplishments  this  year. 

67th  MORSS 

You  should  have  already  received  your 
registration  packets  in  the  mail.  Anne 
Patenaude  and  her  staff  have  done  an  out¬ 
standing  job  preparing  and  organizing  this 
Symposium,  but  it  is  missing  one  main 
ingredient  —  your  participation.  Please 
remember  to  submit  your  application  by 
the  16  April  deadline  for  regular  registra¬ 
tion  fees.  After  that  applications  will  be 
accepted,  but  the  registration  fees  will 
increase.  If  you  haven’t  already  made  your 
hotel  reservations,  open  the  registration 
packet  to  page  seven  and  make  them 
NOW.  Hotel  space  is  hmited  and  there  are 
not  many  campgrounds  nearby. 

New  Sponsors 

I  had  the  pleasure  of  meeting  and  intro¬ 
ducing  MORS  to  our  two  newest  Sponsors. 
They  are  RADM  Raymond  Smith  of  N81 
within  the  OPNAV  staff  and  MaJGen  Ken¬ 
neth  Hess,  who  is  Director  of  Command 


and  Control  within  the  Air  Force  headquar¬ 
ters  staff.  Welcome  aboard! 

Use  of  the  Theme  -  ^Developing 
the  Junior  Analyst” 

The  Education  Colloquium  is  one  of  the 
major  events  of  the  year  on  the  profession¬ 
al  affairs  side  of  the  house.  This  year’s 
colloquium  features  the  junior  analyst. 
Please  take  time  to  read  the  VP(PA)’s  arti¬ 
cle  by  CAPT  Lee  Dick  in  this  edition.  I 
continue  to  see  great  payoffs  in  the  time 
invested  in  the  development  of  our  junior 
analysts. 

Hold  Up  Those  Four  Fingers 

You  will  often  see  players  holding  up 
four  fingers  at  the  beginning  of  the  fourth 
quarter  of  football  games.  This  helps  the 
players  recognize  this  is  the  fourth  and 
final  quarter  of  the  game.  Regardless  of  all 
the  sweat  lost  during  the  first  three  quar¬ 
ters,  you  must  make  that  extra  push  into 
the  last  quarter  to  finish  the  game  a  “win¬ 
ner,”  regardless  of  how  far  ahead  or  behind 
you  are.  We  are  indeed  entering  the  fourth 
quarter  of  the  game  with  much  to  complete 
in  terms  of  meeting  preparation  and  execu¬ 
tion.  We  cannot  be  successful  without 
your  hard  work  and  participation.  Just 
ensure  one  thing  while  you  are  in  the  ball- 
game  —  teach  the  “rookie”  beside  you 
how  to  play  the  game.  Keep  strokin.’  O 


Upcoming  MORS 
Meetings 

Education  Colloquium 

22, 23  April  1999 

ANSER  Headquarters,  Arlington,  VA 

67th  MORS  Symposium 

22, 23, 24  June  1999 
US  Military  Academy 
West  Point,  NY 


Dennis  Baer 

MORS  President 


PHALANX 


3 


March  1999 


MAS  PRESIDENT 


Setting  Goals  on  Our  Educational  Cornerstone 


I  just  returned  from 
the  Army  Simula¬ 
tion  and  Modeling 
for  Acquisition,  Re¬ 
quirements  and  Train¬ 
ing  (SMART)  Confer¬ 
ence  in  San  Antonio. 
Diane  Scharein, 
TRADOC  HQ,  had 
asked  me  to  attend  so  I 
could  support  the 
Requirements  Breakout  Session  she  and 
COL  Mike  Lavine,  Office  of  the  Assistant 
Secretary  of  the  Army  for  Research  Devel¬ 
opment  and  Acquisition  (SARD A),  ran,  and 
be  my  usual  obnoxious  self.  Normally,  I 
tend  to  pass  up  these  management-stake¬ 
holder  conferences  for  the  technical  ones, 
but  that  would  have  been  a  mistake.  The 
information  and  interaction  were  great  and  I 
offer  kudos  to  LTG  Paul  Kern  and  Dr. 
Herb  Fallin.  I  came  away  with  some  tech¬ 
nical  challenges  for  the  profession  that  I 
want  to  share:  Physics  Based  Operations 
Research;  and  Systems  of  Systems  that  fre¬ 
quently  collapse  into  Mega-Systems. 

Before  I  go  any  further,  I  need  to  recog¬ 
nize  Ensign  (then  Cadet)  Jeff  Kuck  who 
received  the  MAS  OR  Student  Award  when 
he  graduated  from  the  US  Coast  Guard 
Academy  in  Operations  Research.  CAPT 
Len  Kelly,  Chair,  Mathematics  Department 
hosted  the  ceremony;  Jim  Malley,  Textron 
Systems  (a  20  year  MASite)  presented  the 
award.  Congratulations  Jeff!  We  have  great 
expectations  of  you.  (For  more  details  on 
all  1998  MAS  Awards  visit  the  MAS  web 
page  at  www.informs.org/subdiv/Soci- 
eties/MAS/index.htm 

In  his  last  message,  my  predecessor  Dr. 
Tom  GuUedge,  George  Mason  University, 
mentioned  that  I  would  be  succeeding  him. 
This  will  definitely  not  be  a  replacement; 
Tom  leaves  some  big  shoes  to  fill  and  time 
will  tell  how  well  this  simple  county  boy 
from  North  Alabama  will  do.  We  owe  Tom 
a  big  debt  for  his  leadership  but  he  will  still 
be  around  as  Past  President  and  INFORMS 
Vice  President,  Meetings  to  keep  us  in  the 
roadway  with  his  wise  counsel.  Dr.  J.  P. 
Ballenger,  Raytheon,  is  our  new  Vice  Pres¬ 
ident  and  President-Elect,  and  Dr,  Philipp 


MAS  OR  Student  Award  winner 
Ensign  JejfKuck 


Djang,  TRAC  White  Sands  Missile  Range 
(WSMR),  is  our  new  Secretary-Treasurer. 
Dr,  Greg  Parnell,  Virginia  Commonwealth 
University,  and  Dr.  Ralph  Toms,  SRI  are 
our  new  Councilors-at-Large  (CAL).  Tom 
GuUedge  also  appointed  Dr,  Pauline 
Cason,  SAIC,  to  serve  out  the  remainder  of 
Philipp’s  term  as  CAL  as  one  of  his  last 
official  acts. 

Our  geographic  chapter,  the  Redstone 
Arsenal  -  Huntsville  Military  Operations 
Research  Section  (RAHMORS)  also 
installed  new  officers.  The  Past  President, 
Dr.  Jeff  Cemy,  AMCOM,  was  elevated  to 
the  Executive  Council  and  after  his  four 
year  term  will  automatically  become  a 
Graybeard  of  the  Section  (GS),  which  is  fit¬ 
ting  since  his  beard  is  showing  a  lot  of  gray 
these  days.  Ray  Livingston,  TBE,  moved 
on  to  become  Past  President,  and  was  suc¬ 
ceeded  by  Hugh  Griffis,  Signature 
Research,  as  President.  Sam  McNulty, 
SAIC,  was  installed  as  Vice  President,  Mrs. 
Martha  Knott,  was  returned  as  Secretary, 
and  Dr.  John  Hall,  TASC  was  installed  as 
Treasurer.  RAHMORS  has  some  good  ini¬ 


tiatives  going  and  I  will  try  to  share  them  in 
later  columns.  One  of  Hugh’s  first  acts  was 
to  present  me  the  first  RAHMORS  Light 
Sabre,  which  I  am  sure  is  destined  be  highly 
coveted. 

One  of  my  first  acts  as  President  was  to 
institute  the  Presidents  Award,  This  award 
consisting  of  a  medal  and  a  certificate,  is  to 
be  presented  to  anyone  who,  in  the  Presi¬ 
dent’s  view,  has  significantly  contributed  to 
the  advancement  and  wellbeing  of  the  Soci¬ 
ety  or  the  Profession.  This  is  an  “as 
required”  rather  than  an  annual  award  and  is 
awarded  at  the  discretion  of  the  President. 
During  our  business  meeting  at  the  Seattle 
INFORMS  Meeting,  I  presented  President’s 
Awards  to  Ray  Livingston,  TBE,  for  his 
service  as  Chief  of  Staff  of  our  first  inde¬ 
pendent  national  meeting,  IMAS,  Pat 
Murphy,  CAA,  for  service  as  Track  Chair 
of  the  Seattle  meeting,  Dr.  Dean  Hartley, 
ORNL,  for  his  service  as  President  and 
Division  Representative,  Dr,  Steve  Balut, 
IDA,  for  his  service  as  President  and  Editor 
of  Topics  in  OR,  and  Dr.  Tom  GuUedge, 
GMU,  for  his  service  as  President.  Subse¬ 
quently,  I  also  presented  a  President’s 
Award  to  COL  Tony  Brinkley,  (Ret)  GS, 
Vitronics  (then  TBE),  for  his  service  as 
General  Chair  of  IMAS.  I  solicit  your 
nominations  of  deserving  recipients  of  this 
award. 

A  good  president  is  supposed  to  come 
into  office  with  a  set  of  goals  and  objec¬ 
tives,  a  program,  if  you  will.  One  of  my 
goals  is  to  increase  opportunity  for  profes¬ 
sional  development.  We  have  a  strong 
foundation  in  our  technical  track  at  the 
INFORMS  national  meeting,  our  annual 
national  meeting  and  the  Topics  in  Opera¬ 
tions  Research.  A  lot  may  be  done  how¬ 
ever,  to  improve  both  individual  and  collec¬ 
tive  educational  opportunities.  The  service 
schools  and  some  of  the  universities  provide 
a  great  deal,  but  not  everyone  can  attend  the 
Naval  Postgraduate  School  or  AFIT.  I 
would  like  to  build  on  this  by  expanding  the 
number  and  scope  of  available  reference 
and  text  materials,  and  classroom  opportu¬ 
nities,  by  developing  both  model  courses 
for  export  to  universities  and  actual  course- 
(See  MAS  PRESmENT,  p.  27) 


Dr.  Bruce  Fowler 
MAS  President 
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VEEPS  PEEP 


Junior  Analysts  —  You  Can  Play  a  Part 


As  we  continue 
to  expound 
on  the  1999 
MORS  theme  of 
“Developing  the 
Junior  Analyst,”  Fve 
issued  some  chal¬ 
lenges  for  you  with¬ 
in  each  of  the  Pro¬ 
fessional  Affairs 
committees.  The 
care  and  feeding  of  young  analysts  is  not 
solely  the  responsibility  of  our  education¬ 
al  institutions,  nor  the  Society  as  a  whole. 
But  rather  it  more  importantly  hinges  on 
the  day-to-day  direct  contact  that  each  of 
us  in  supervisory  or  senior  peer  positions 
play  in  developing  the  techniques  and  the 
allegiance  that  will  one  day  make  them 
leaders  in  our  field. 

Education  and  Professional 
Development  Committee 

The  premier  Junior  Analyst  event  of 
the  year  is  fast  approaching  next  month. 
If  you  have  not  already  made  plans  to 
send  your  junior  analysts  to  the  Education 
Colloquium  on  22-23  April  at  ANSER 
HQ  in  Crystal  City,  please  mark  the  dates 
on  their  calendars  and  try  to  get  them 
interested  in  attending.  I  would  like  to 
challenge  each  supervisor  to  ensure  that 
your  office  is  represented  by  at  least  one 
junior  analyst  at  the  99  Colloquium.  I 
have  passed  the  information  along  to  the 
entire  Director  of  Naval  Training  staff, 
with  the  goal  of  not  only  bringing  our 
three  Operations  Analyst  (OA)  graduates 
to  the  event,  but  the  majority  of  the  staff 
as  well.  As  the  purveyors  of  training  and 
education  in  the  Navy,  it  will  be  a  great 
opportunity  to  raise  their  level  of  analyti¬ 
cal  awareness,  as  well  as  providing  some 
experts  in  training  and  education  to  dia¬ 
logue  with  our  analysts  in  attendance.  A 
special  thanks  goes  out  to  Major  Willie 
McFadden,  chairmen  of  the  Education 
and  Professional  Development  Commit¬ 
tee  for  the  effort  he  has  put  into  the  plan¬ 
ning  of  this  year’s  activity.  Other  com¬ 
mittee  members  who  deserve  credit  are 
Tutorials  Coordinator  Dean  Hartley, 


Junior  Analyst  Coordinator  Howard 
Whitley  and  Panel  Discussion/Special 
Presentations  Coordinator  Col  “Crash” 
Konwin.  The  keynote  speaker  for  this 
year’s  colloquium  will  be  Mr.  Walt  Hol¬ 
lis,  DUSA(OR),  FS.  Please  check  the 
MORS  homepage  http://www.mors.org 
frequently  for  updates. 

Prize  Committee 

So  what  do  your  junior  analysts  know 
about  the  MORS  prize  competitions  for 
technical  accomplishments?  Are  they 
aware  of  the  $1000  Barchi  Prize,  the 
“Best  of  the  Best”  Prize,  which  is  award¬ 
ed  to  the  best  paper  resulting  from  the 
best  presentation  in  each  working  or  com¬ 
posite  group  at  the  previous  Symposium? 
Do  they  know  about  the  $1000  Rist  Prize 
which  is  awarded  each  year  to  the  best 
paper  received  in  response  to  an  annual 
call  for  papers?  This  year,  MORS 
received  24  papers  to  compete  for  the 
Barchi  Prize  and  18  papers  to  compete  for 
the  Rist  Prize.  The  quantity  and  quality  of 
this  year’s  competition  is  exceptionally 
good.  According  to  Prize  Committee 
Chair  Maj  Mark  Gallagher,  we  expect 
more  prize  competition  papers  will  be 
published  in  Military  Operations 
Research  than  any  previous  year.  At  the 
June  Symposium,  both  the  Barchi  and 
Rist  winners  will  present  their  work  dur¬ 
ing  a  special  session.  In  addition,  the 
winners  and  those  individuals  receiving 
honorable  mentions  will  also  be  asked  to 
display  their  work  during  the  Tuesday 
evening  mixer. 

Heritage  Committee 

Have  you  introduced  your  junior  ana¬ 
lysts  to  a  MORS  classic  this  year?  Why 
not  plan  to  present  the  top  analyst  in  your 
shop  with  a  copy  of  the  next  MORS 
reprint  when  it  becomes  available  later 
this  year?  B.  O.  Koopman’s  book  on 
Search  and  Screening  was  nominated  as 
the  next  volume  in  the  MORS  Classics 
series.  The  Heritage  Committee  Chair 
Mr.  Ted  Smyth  has  obtained  a  copy  of 
the  book  and  provided  it  to  the  MORS 
staff  which  is  in  the  process  of  contacting 


the  publisher  to  discuss  copyrights  and 
other  legal  issues. 

The  MORS  Oral  History  project  will 
be  a  great  way  to  enable  our  junior  ana¬ 
lysts  to  learn  about  OR  roots.  Another 
oral  history  interview  has  been  conduct¬ 
ed,  this  time  with  Mr.  Clay  Thomas,  FS, 
Chief  Scientist,  US  Air  Force  Studies  and 
Analysis  Agency.  The  interview  was  con¬ 
ducted  by  Mr.  Gene  Visco,  FS,  with 
assistance  from  Dr.  Bob  Sheldon.  The 
tape  of  the  interview  has  been  transcribed 
and  is  in  the  process  of  being  edited.  We 
plan  to  have  an  article  on  Mr.  Thomas, 
with  accompanying  portions  of  the  inter¬ 
view,  appear  in  the  spring  edition  of  the 
MOR  journal.  In  addition  we  hope  to 
have  an  article  in  the  PHALANX  prior  to 
the  67th  MORSS  and  the  award  of  the 
first  Thomas  Prize. 

Publications  Committee 

Do  your  junior  analysts  regularly  read 
copies  of  the  PHALANX  and  the  MOR 
journal?  Are  they  available  to  everyone 
in  your  shop?  Have  you  told  them  about 
PHALANX  Online  and  did  they  visit  the 
web  site  to  read  the  latest  edition?  The 
dissemination  of  information  within  the 
Society  is  key  to  maturing  new  analysts 
and  in  keeping  our  Society  healthy  and 
stimulated.  As  we  continue  to  focus  on 
the  MOR  analysts  and  the  decision  mak¬ 
ers  that  use  MOR  analysis,  the  high  quali¬ 
ty  and  relevance  of  both  MOR  and  PHA¬ 
LANX  must  be  continued.  We  all 
recognize  the  importance  of  timely  publi¬ 
cation  of  special  meeting  reports.  The 
Publications  Committee,  under  the  lead¬ 
ership  of  Mr.  Brian  Engler,  and  the 
MORS  staff  have  been  focused  on  clear¬ 
ing  out  the  several-year  Special  meetings 
publication  backlog.  The  Publications 
Committee,  in  coordination  with  the  Spe¬ 
cial  Meetings  Committee,  will  continue  to 
stress  the  timely  production  of  Special 
Meetings  publications.  To  that  end,  a 
joint  sub-committee,  chaired  by  Dr.  Stu¬ 
art  Starr,  FS,  is  examining  special  meet¬ 
ing  publication  procedures  to  see  if  they 
can  be  further  streamlined.  O 
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System  Stability:  A  Proxy  for  “Graceful  Degradation” 

Dr,  James  P.  Ignizio,  Ph.D.,  US  Army  Logistics  Management  College 
e-mail:  ignizio@virginia.edu 


Within  the 
military 
sector  the 
notion  of  “graceful 
degradation’’  is  uni¬ 
versally  accepted. 
Mihtary  systems  (e.g., 
weapons,  force  mixes, 
communication  links, 
air  defense  systems 
and  even  a  “system” 
of  strategies  and  tactics)  should,  it  is 
agreed,  gracefully  degrade  (e.g.,  under 
hostile  conditions,  or  random  failures,  or 
variations  in  mission,  or  changes/modifica¬ 
tions  in  personnel  and  equipment)  — 
rather  than  collapse  like  a  house  of  cards. 

Unfortunately,  there  is  no  agreement  as 
to  how  one  defines  graceful  degradation^ 
or  how  it  is  measured.  Furthermore,  and 
perhaps  most  unsettling,  the  attributes  of 
optimality  and  graceful  degradation  may 
—  if  the  hypothesis  of  this  article  holds  — 
actually  be  in  opposition.  To  illustrate  this 
phenomenon,  consider  the  simple  “block 
world”  problem  depicted  in  Figure  1 . 

The  stack  of  blocks  on  the  left  side  of 
Figure  1  is  unequivocally  optimal  in  the 
sense  of  being  the  taller  of  the  two  stacks. 
However,  while  the  stack  on  the  right  side 
of  the  figure  is  shorter,  less  impressive  in 
appearance,  and  “sub-optimal;”  it  is  also 
clearly  far  more  stable.  Given  the  choice 
between  attempting  to  stand  on  either  of 
the  two  stacks,  most  people  would  select 
the  suboptimal  stack.  Clearly,  something 
more  than  the  height  of  the  stack  is  impor¬ 
tant  —  something  difficult  to  put  into 
words  or  formulas. 

In  this  article  I  explore  the  very  real 
possibihty  that  “optimal”  solutions  may  be 
invariably  unstable  —  wherein  stability  is 
defined  as:  ''the  measure  of  both  the  speed 
and  ease  by  which  a  given  solution  ^de- 
evolves*  (degrades)  to  some  minimally 
acceptable  level  ”  In  the  case  of  the  "block 
world''  illustration  given  earlier,  it  should 
be  apparent  that  the  “optimal”  stack  is  like¬ 
ly  to  collapse  easier  and  faster  than  the 
shorter  stack. 

In  addition,  what  would  appear  (based 
on  results  thus  far)  to  be  a  practical  and 
effective  approach  for  the  assessment  of 


the  stability  of  any  given  solution  is  pre¬ 
sented.  Its  performance  on  a  number  of  real 
world  problems  is  described.  I  then  con¬ 
trast  the  inherent  stability  of  solutions  as 
produced  by  traditional  optimization  with 
those  developed  by  evolutionary  means 
(e.g.,  genetic  algorithms,  evolutionary 
computation). 

While  the  results  of  my  investigation 
have  thus  far  upheld  my  hypothesis  (i.e., 
that  unstable  solutions  de-evolve  faster  and 
easier  than  stable  ones),  it  should  be  made 
clear  that  these  results  have  been  limited  to 
the  (intensive)  investigation  of  but  nine 
problems  (albeit  real  problems  and  real 
data).  Since  there  would  appear  to  be  no 
way  to  investigate  the  phenomenon  of  sta¬ 
bility  other  than  empirically,  it  is  hoped 
that  this  article  motivates  others  to  evaluate 
the  process  on  their  own  set  of  (real  world) 
problems. 

Conventional  wisdom  holds  that  one 
should  always  strive  for  solutions  that  are 
optimal,  or  at  least  “near  optimal.”  The 
idea  of  intentionally  developing  non-opti- 
mal  solutions  is,  in  itself,  an  anathema  to 
the  operations  research  profession  —  and 
particularly  to  the  academic  community. 
However,  as  a  practicing  OR  analyst  for 
more  than  30  years,  I  have  noted  that  a  sur¬ 
prising  number  of  “optimal  solutions”  to 


real  world  problems  have  led  to  unexpect¬ 
ed  and  troubling  consequences.  Specifical¬ 
ly,  while  such  solutions  may  be  optimal  on 
paper,  they  prove  to  be  problematic  when 
actually  implemented. 

Just  two  of  the  many  indications  of 
instability  of  optimal  solutions  I  have  per¬ 
sonally  encountered  are  listed  below: 

SAM-D:  SAM-D  was  the  original  des¬ 
ignation  for  what  is  now  known  as  the 
Patriot  air  defense  system.  In  the  late  nine¬ 
teen-sixties  I  was  tasked  with  the  develop¬ 
ment  of  a  scheme  for  the  deployment  of 
the  elements  of  such  a  system.  In  other 
words,  to  produce  a  method  to  locate  the 
sites  for  the  missile  launchers  and  radars  so 
as  to  minimize  “leakage”  (i.e.,  protect  a 
region  of  airspace  from  attack  by  enemy 
aircraft).  It  was  discovered  that  a  branch- 
and-bound  approach,  which  guaranteed 
optimal  or  near  optimal  solutions,  also 
resulted  in  deployments  that  were  extreme¬ 
ly  unstable.  For  example,  if  some  combina¬ 
tion  of  system  elements  (e.g.,  positions, 
weather,  target  signature)  were  changed  — 
even  slightly,  air  defense  performance 
would  often  suffer  a  dramatic  reduction. 
Yet,  when  deployed  by  means  of  a  heuris¬ 
tic  method,  the  results  were  quite  stable  — 
at  the  cost  of  but  a  very  slight  reduction  in 
the  “optimality”  of  the  solution. 


Figure  1:  “Optimal”  and  “Suboptimal”  Stacks 


James  P.  Ignizio, 
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Torpedo  Acoustic  Arrays:  Acoustic 
arrays  for  torpedoes  (or  electromagnetic 
arrays  for  radars)  consist  of  a  number  of 
transducers,  acting  as  an  ensemble.  These 
transducers  are  to  be  located  in  such  a 
manner,  and  delivered  power  of  such 
amplitude  and  phase,  as  to  produce  a  beam 
pattern  of  a  specified  shape.  In  essence,  the 
array  design  problem  is  one  of  combinator¬ 
ial  optimization,  and  may  be  formulated 
and  solved  by  conventional  methods  for  an 
optimal  solution.  However,  once  these 
“optimal”  acoustic  arrays  are  constructed 
and  tested  (typically  in  a  water  chamber, 
under  a  variety  of  conditions),  the  actual 
performance  may  be,  in  a  word,  awful  Just 
a  few  seemingly  insignificant  changes  in  a 
combination  of  parameters  (e.g.,  a  slight 
decrease  in  temperature  coupled  with  a 
small  increase  in  pressure)  can,  and  often 
does,  result  in  sudden  and  dramatic  degra¬ 
dation  of  the  beam  pattern.  As  such,  a 
design  that  is  optimal  on  paper  may  well  be 
impractical  for  application. 

While  numerous  other  instances  could 
be  cited,  all  have  a  similar  property.  That 
is,  optimal  solutions,  even  when  accompa¬ 
nied  by  intensive  (but  conventional)  sensi¬ 
tivity  analysis,  are  often  found  to  be  highly 
unstable.  Yet  solutions  that  are  heuristical- 
ly  derived,  and  clearly  sub-optimal,  can  be 
as  “solid  as  a  rock.” 

Based  upon  these  experiences,  coupled 
with  a  long-time  interest  in  heuristic  meth¬ 
ods  (particularly  those  that  mimic  evolu¬ 
tion,  such  as  genetic  algorithms),  I  sought 
to  test  the  following  hypothesis: 

Inherently  unstable  solutions  will  de- 
evolve  faster  and  easier  than  stable  solu¬ 
tions. 

To  determine  the  validity  of  this 
hypothesis,  a  “reverse”  genetic  algorithm 
was  developed.  That  is,  instead  of  starting 
from  a  poor  (or  randomly  selected)  solu¬ 
tion  and  seeking  to  evolve  to  more  fit  solu¬ 
tions,  my  algorithm  begins  with  any  given 
solution  (e.g.,  the  solution  to  be  tested  for 
stability)  and  de-evolves  to  less  fit  solu¬ 
tions.  It  was  (as  implied  in  the  hypothesis) 
my  conjecture  that  an  unstable  solution 
would  de-evolve  (e.g.,  collapse)  in  fewer 
generations  than  a  stable  one. 

For  those  unfamiliar  with  genetic  algo¬ 
rithms,  a  list  of  resources  is  provided  at  the 
end  of  this  article. 

In  brief,  a  genetic  algorithm  proceeds 
by  first  coding  a  trial  solution  into  a  “chro¬ 
mosome”  (e.g.,  a  pattern  of  zeros  and  ones 
that  serve  to  represent  the  values  of  the 


decision  variables).  Next,  a  population  of 
solutions  (chromosomes)  is  generated  (typ¬ 
ically  randomly).  From  there,  a  parallel  and 
probabilistic  search  procedure  ensues. 
Solutions  are  evaluated  for  their  “fitness.” 
Those  that  are  most  fit  are  given  a  higher 
likelihood  of  being  placed  into  a  “mating 
pool.”  The  mating  pool  is  generated,  sto¬ 
chastically,  and  solutions  “exchange  por¬ 
tions  of  their  chromosomes”  with  their 
mates  so  as  to  produce  new  solutions  (e.g., 
an  exchange  of  a  segment  of  zeros  and 
ones  in  one  chromosome,  or  coding,  with 
those  in  another).  Mutation  (e.g,,  the  “flip¬ 
ping”  of  a  zero  to  a  one,  or  vice-versa)  then 
takes  place  (albeit  with  a  very  low  proba¬ 
bility)  and  the  resulting  set  of  solutions 
represent  the  “next  generation,”  The 
process  repeats  until  a  given  termination 
criterion  is  reached. 

Since  my  intent  is  to  find  out  how  easy 
and  fast  a  given  solution  de-evolves  (rather 
than  evolves),  my  genetic  algorithm  starts 
with  the  solution  (chromosome)  to  be  test¬ 
ed  and  works  backwards.  The  pseudo-code 
for  the  de-evolution  algorithm  is  provided 
below. 


procedure  De-Evolve: 

begin 

t=0 

select  chromosome,  C(t) 
perturb  C(t)  [to  generate  initial 
population,  P(t)] 
fitness  P(t) 
until  (done) 
t  =  t  +  1 

select  P(t)  from  P(t-l) 
crossover  P(t) 
mutate  P(t) 
fitness  P(t) 
end 


It  is  probably  easiest  to  explain  the 
process,  and  its  interpretation,  by  example. 
Thus,  first  consider  the  results  shown  in 
Figure  2.  In  this  figure,  two  solutions  to  the 
design  of  the  “backbone”  network  for  a 
telecommunications  system  are  depicted. 
The  diagnol  symbol,  to  the  far  left  of  the 
figure,  depicts  the  optimal  solution  to  the 
problem  (producing  a  normalized  value  of 
100  for  the  measure  of  messages  per  unit 
time).  The  box  symbol  on  the  far  left  repre¬ 
sents  a  solution  derived  via  a  genetic  algo¬ 
rithm.  Its  value  is  94  units,  some  6  percent 
less  than  that  of  the  optimal  solution.  Let 


us  assume  that  the  minimally  acceptable 
level  of  system  performance  is  60  units. 
Using  the  de-evolution  algorithm,  we  then 
determine  just  how  many  generations  are 
required  for  both  solutions  to  degrade  to 
the  level  of  60  message  units. 

Examining  Figure  2,  we  see  that  the 
“sub-optimal”  solution  takes  roughly 
twice  as  long  (more  than  20  generations) 
to  degrade  to  the  minimally  acceptable 
level  than  does  the  optimal  solution 
(which  de-evolves  to  the  minimally 
acceptable  level  in  just  10  generations).  In 
other  words,  the  sub-optimal  solution  is 
apparently  more  stable  than  the  optimal 
solution. 

Of  course,  the  result  shown  in  Figure  2 
might  just  be  a  fluke.  After  all,  we  are 
dealing  with  a  stochastic  search  process. 
Consequently,  the  de-evolution  algorithm 
is  repeated  numerous  times  (using  differ¬ 
ent  random  number  seeds)  and  the  results 
presented  in  a  histogram  like  that  shown 
in  Figure  3. 

In  Figure  3  it  is  apparent  that  the  opti¬ 
mal  solution  (i.e.,  the  de-evolutions  shown 
to  the  left  of  the  vertical  dashed  hne)  does 
in  general  de-evolve  faster  and  easier  than 
the  suboptimal  solution  (those  to  the  right 
of  the  dashed  line,  as  originally  derived  by 
means  of  a  genetic  algorithm).  Results  of 
many  more  de-evolutions,  as  well  as  the 
investigation  of  some  eight  other  real 
world  problems  confirmed  this  observa¬ 
tion.  One  of  these  eight  problems  was  that 
of  the  siting  of  the  elements  of  the  Patriot 
Air  Defense  System. 

Using  data  from  the  original  SAM-D 
air  defense  study,  I  developed  a  number  of 
different  siting  schemes  (e.g.,  location 
coordinates  for  the  radars  and  missile 
launchers)  for  the  air  defense  system.  One 
of  the  solutions  was  optimal,  having  been 
derived  via  a  tedious  and  time  consuming 
implicit  enumeration  method.  Another 
solution  was  derived  by  means  of  a  genet¬ 
ic  algorithm.  All  other  solutions  (eight 
more  in  total)  were  developed  by  various 
perturbations  of  these  two  results.  The 
problem  parameters  were  that  of  the  pre¬ 
cise  coordinates  where  each  element  of 
the  air  defense  systems  was  located,  the 
estimates  of  terrain  topology  and  weather 
parameters.  The  fitness  of  the  solution  was 
based  upon  the  amount  of  airspace  cov¬ 
ered  by  the  air  defense  system.  The  results 
are  shown  in  Table  1. 

(See  STABILITY,  p.  8) 
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STABILITY 

(continued from  p.  7) 

Examining  Table  1,  we  note  that  the 
global  optimal  solution  is  air  defense  sys¬ 
tem  A;  the  one  developed  via  implicit 
enumeration.  Solution  F  is  the  solution 
developed  by  means  of  a  genetic  algo¬ 
rithm.  Not  only  is  solution  F  not  optimal, 
it  is  actually  dominated  by  solution  C. 
That  is,  C  is  both  cheaper  and  has  a  high¬ 
er  measure  of  effectiveness  than  F.  If  we 
were  to  stop  our  analysis  at  this  point, 
solution  F  wouldn’t  look  very  attractive 
at  all. 

However,  after  applying  the  de-evolu¬ 
tion  algorithm  to  all  ten  solutions  (and 
after  repeating  the  process  numerous 
times,  using  different  random  number 
schemes),  it  was  found  that  solution  F 
was,  far  and  away,  the  most  stable  solu¬ 
tion  of  all  those  tested.  Solution  A,  the 
“optimal”  scheme,  fell  apart  with  only 
slight  changes  in  various  combinations  of 
model  parameters. 

Thus  we  are  left  with  the  following 
choice.  We  may  either  pick  the  global 
“optimal”  solution  (or  any  solution  with  a 
higher  effectiveness  to  cost  ratio  than 
solution  F),  and  suffer  the  consequences 
of  moderate  to  extreme  instability,  or 
select  the  rock-steady  solution  F  —  at  a 
slight  reduction  in  the  efficiency  to  cost 
ratio.  When  one  considers  the  fact  that 
the  minor  difference  in  effectiveness  to 
cost  could  well  be  a  result  of  errors  in 
data  (and  all  real  world  problems  have 
such  errors),  solution  F  starts  looking 
very  attractive. 

As  mentioned  earlier,  results  on  nine 
different,  real  world  problems,  have  all 
substantiated  my  original  hypothesis. 
Does  this  mean  that  optimal  solutions  are 
always  unstable?  Or  that  solutions 
derived  by  genetic  algorithms  are  always 
more  stable  than  optimal  solutions? 

The  short  answer  is  no.  Just  nine 
experiments  obviously  cannot  prove  or 
disprove  the  hypothesis.  However,  these 
nine,  highly  consistent  results  should 
make  the  OR  community  take  pause. 
Hopefully  this  article  will  encourage  oth¬ 
ers  to  investigate  this  matter  also.  The 
more  empirical  evidence  in  support  of 
this  article’s  hypothesis,  the  more  wary 
any  OR  practitioner  should  be  of  unques¬ 
tionably  accepting  the  doctrine  of  “opti¬ 
mality.” 

STABILITY,  p.  27) 


Figure  2:  The  de-evolution  of  two  different  solutions  to  a  network  design  problem 


Figure  3:  Histogram  of  De-evolutions 


Candidate  System 

Cost 

Effectiveness 

Effectiveness  to  cost  ratio 

A 

50 

80 

1.6 

B 

48 

76.32 

1.59 

C 

52 

84.42 

1.585 

D 

54 

84.78 

1.57 

E 

49 

76.44 

1.56 

F 

53 

81.62 

1.54 

G 

56 

84 

1.5 

H 

57 

79.8 

1.4 

I 

56 

67.2 

1.2 

J 

59 

64.9 

1.1 

Table  1:  Candidate  Air  Defense  Systems 
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Introduction 

The  21st  century 
promises  the 
availability  of 
advanced  defense  tech¬ 
nologies  that  will  revo¬ 
lutionize  Command, 
Control,  Communica¬ 
tions,  Computers, 
Intelligence,  Surveil¬ 
lance  and  Reconnais¬ 
sance  (C^ISR)  activi¬ 
ties.  There  are  systems  being  designed  and 
built  to  support  military  decision  making  in 
deliberate  command  and  control  activities, 
and  to  automate  the  “sensor-to-shooter” 
linkages;  making  precision  fires  more  lethal. 
Joint  Vision  2010  prescribes  an  “operational 
template”  that  capitalizes  on  these  emerging 
technologies.  1  Many  military  experts  assert 
that  these  technologies  and  the  doctrine  they 
support  constitute  a  Revolution  in  Military 
Affairs  (RMA).  The  belief  is  that  this  revo¬ 
lution  will  provide  the  United  States’  mili¬ 
tary  the  ability  to  dominate  any  projected 
opponent  on  the  battlefield  of  the  future. 

The  military  operations  research  commu¬ 
nity  is  a  key  part  of  the  intellectual  effort 
supporting  the  asserted  RMA.  Computer- 
based  simulation  results  are  used  as  key  evi¬ 
dence  of  both  individual  system  effective- 
ness  and  improvements  in  “force 
effectiveness.”  The  operational  testing 
community  assures  military  leadership  that 
the  systems  the  Department  of  Defense 
(DoD)  plans  on  acquiring  meet  the  specified 
operational  “requirements.”  Analysts  sup¬ 
port  warfighting  experiments  and  collect 
data  that  “prove  on  the  ground”  the  success 
of  the  RMA.  We,  in  the  military  operations 
research  community,  are  the  purveyors  of 
quantifiable  evidence  that  demonstrates  the 
increasing  effectiveness  of  the  US  military. 

There  is  also  a  significant  intellectual 
effort  that  is  seeking  to  identify  new  Mea¬ 
sures  of  Effectiveness  (MOE).  That  effort 
seems  to  assume  that  the  hypothesis  has 
been  accepted;  that  the  RMA  is  a  proven 
replacement  for  existing  capabilities  and 
doctrine.  There  also  seems  to  be  an 
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assumption  that  the  underlying  objectives  of 
warfare  are  changing.  Yet,  there  are  histori¬ 
cal  examples  that  indicate  we  should  be 
very  cautious  in  drawing  such  sweeping 
conclusions  about  military  effectiveness  or 
model  based  analysis. 

One  example  of  misinformed  analysis 
occurred  between  World  War  I  and  World 
War  II  in  France.  The  French  military 
invested  heavily  in  the  creation  of  the  Mag- 
inot  Line.  These  significant  investments 
were  complemented  by  a  carefully  crafted 
and  rigorously  analyzed  doctrine  that 
emphasized  coordinated,  superior  fires. 
However,  the  German’s  successful  cam¬ 
paign  around  the  line  is  unequivocal  evi¬ 
dence  that  this  strategy  was  not  only  not 
revolutionary,  but  a  failure.^  There  is  also 
relatively  recent  evidence  outside  military 
history  indicating  that  analytically  supported 
“certainty”  can  have  unexpected  and  poten¬ 
tially  devastating  consequences.  For  exam- 
ple: 

•  On  28  January  1986,  the  space  shuttle 
Challenger  exploded  over  the  Atlantic 
Ocean  almost  immediately  after  launch. 
A  retrospective  analysis  of  the  decision 
process  leading  up  to  the  launch  indicat¬ 
ed  that  the  analysis  supporting  NASA 
leadership  was  flawed.  The  information 
presented  to  decision  makers  regarding  a 
potential  “O”  ring  failure  did  not  high¬ 
light  the  impact  of  cold  temperature  on 
the  part’s  reliability.  Had  temperature 
been  highlighted  in  the  analysis,  the  like¬ 
lihood  of  “O”  ring  failure  would  have 
been  identified  as  being  very  high.^ 

•  In  1979  the  Three  Mile  Island  nuclear 
reactor  released  hazardous  radiation  into 
the  atmosphere.  The  risk  analysis  that 
was  provided  to  the  Nuclear  Regulatory 
Commission  to  support  licensing  deci¬ 
sions  demonstrated  the  facility’s  safety.^ 

Joint  Vision  2010  is  similar  to  the 
French  experience  in  that  there  has  been  a 
tremendous  resource  investment  and  it  is 
supported  by  extensive  analysis.  The  non¬ 
military  examples  are  similar  to  the  DoD’s 
current  decision  support  efforts  in  that  they 


are  largely  based  on  computer-based  simu¬ 
lation  and  mathematical  modeling.  The 
conclusion  that  can  be  drawn  from  all  of 
these  examples  is  that  a  nonzero  probability 
exists  (i.e.  it  is  possible)  that  JV  2010’s 
envisioned  capabilities  could  fail  to  produce 
the  intended  results.  Moreover,  no  amount 
of  analysis  or  investment  can  guarantee  suc¬ 
cess.  Properly  framed  and  conducted  analy¬ 
sis  can,  however,  minimize  the  likelihood  of 
a  catastrophic  failure,  and  improve  the 
potential  for  good  outcomes  in  military 
operations. 

This  article  presents  a  foundation  for 
ensuring  that  analysis  (operations  research) 
can  help  focus  the  analytic  community. 
There  are  three  components: 

1)  Answer  the  right  question.  Models  and 
analysis  (and  by  implication  MOE)  must 
be  relevant  and  anchored  to  the  objec¬ 
tives  that  are  appropriate  for  the  decision 
space  under  consideration.  Most  models 
quantify  these  objectives  in  some  man¬ 
ner.  We  will  refer  to  them  as  “funda¬ 
mental  objectives.”^  The  set  of  funda¬ 
mental  objectives  describes  a  decision 
maker’s  essential  reasons  for  being  inter¬ 
ested  in  the  decision  at  hand. 

2)  Make  significant  cause  and  effect  rela¬ 
tionships  transparent.  Modeling  and 
analysis  in  support  of  decision  making 
are  most  useful  when  insight  into  a  com¬ 
plex  decision  situation  is  achieved.  This 
insight  requires  that  the  decision  (option 
space)  be  clear,  and  that  there  be  a  clear 
causal  connection  between  decisions  and 
the  relevant  fundamental  objectives.  In 
complex  models  it  is  often  necessary  to 
create  intermediate  objectives,  which  we 
will  refer  to  as  “means  objectives,”^  to 
illuminate  causal  relationships.  Means 
objectives  derive  their  importance  from 
their  connection  to  other,  more  funda¬ 
mental,  objectives. 

3)  Acknowledge  and  address  uncertainty. 
Uncertainties  exist  with  respect  to  future 
scenarios,  future  weapon  systems  effec¬ 
tiveness  and  threat  behavior.  These 

(See  REVOLUTIONS,  p.  JO) 
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Identifying  and  Organizing 
Objectives  for  Anaiysis 
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Figure  1:  Interaction  between  policy  and  analysis. 


REVOLUTIONS 

(continued from  p.  9) 

uncertainties  must  be  explored  to  evalu¬ 
ate  the  robustness  of  potential  choices  in 

a  decision  situation. 

These  three  concepts  apply  to  all  types 
of  analysis.  In  fact,  they  are  embedded 
somewhere  in  virtually  every  operations 
research  textbook.  This  article  focuses  on 
issues  and  examples  that  are  most  relevant 
to  campaign  level  planning  and  DoD  level 
resource  allocation  decisions.  Each  of  the 
three  concepts  is  addressed  in  the  follow¬ 
ing  sections  and  some  conclusions  are 
offered  in  the  final  section. 

Fundamental  Objectives 

The  “overarching”  measures  of  effec¬ 
tiveness  that  are  applied  to  support  US  mil¬ 
itary  and  political  leaders  must  possess 
some  essential  properties  if  they  are  to  fos¬ 
ter  better  decisions.  First,  they  must  clear¬ 
ly  connect  decision  makers’  concerns  to  the 
analytical  frame  of  reference.  Figure  1 
identifies  two  differing  views  of  the 
defense  decision  environment.  The  top 
half  is  the  view  of  a  policymaker  or  mili- 
taiy  leader.  Policymakers  think  in  terms  of 
strategic  objectives.  Military  leaders  tend 
to  think  in  terms  of  military  success  in  con¬ 
flict.  These  broad  goals  tend  to  be  difficult 
to  describe  and  virtually  impossible  to 
quantify  in  a  “raw”  state. 

The  bottom  half  of  Figure  1  displays  the 
perspective  of  an  analyst.  Analysts  must 
decompose  the  amorphous  goals  into  sets 
of  measurable  objectives  that  capture  the 
essence  of  the  decision  situation,  usually 
implicitly  connected  to  a  set  of  alternatives 
that  are  of  interest  to  decision  makers. 
There  are  a  number  of  paradigms  used  in 
analytic  practice  today  to  describe  the 
essential  elements  of  an  analyst’s  activities. 
Keeney  introduces  a  set  of  concepts  he 
calls  “Value  Focused  Thinking.”^  It  is  the 
vocabulary  that  we  use  in  this  discussion. 
He  uses  the  term  “fundamental  objectives” 
to  describe  a  decision  maker’s  essential 
reasons  for  being  interested  in  a  decision 
situation.  Fundamental  objectives  are 
quantifiable  by  definition,  and  are  normally 
organized  hierarchically.  Each  level  of  the 
hierarchy  is  a  complete  nonoverlapping 
description  of  objective  above  it,  and  the 
uppermost  fundamental  objective  is 
referred  to  as  the  “overall  fundamental 
objective.”  Measurements  associated  with 
the  set  of  fundamental  objectives  can  be 


combined  via  utility  functions  to  express 
an  overall  satisfaction  with  a  course  of 
action.  These  utility  scores  can  then  be 
used  to  support  comparative  analysis. 

In  the  context  of  national  defense, 
America’s  strategic  objectives  are  very  sta¬ 
ble.  A  flag  officer  or  senior  civilian  leader 
could  look  to  joint  military  doctrine  and 
extract  something  like  the  following  to 
articulate  his  or  her  policy-oriented  under¬ 
standing  of  the  objectives:  “National  mili¬ 
tary  strategy  is  to  promote  peace,  deter 
aggression,  and,  failing  that,  fight  and 
win.”^  This  broad  strategic  statement 
might  be  refined  into  the  following  set  of 
objectives: 

•  Accomplish  the  assigned  military 
mission.  This  is  the  military’s  ability  to 
restore  or  create  a  situation  that  is  in  the 
United  States’  interest.  This  must  be 
evaluated  (measured)  in  the  context  of  a 
scenario,  or  some  set  of  scenarios. 

•  Minimize  casualties.  Our  culture  values 
human  life.  Therefore,  the  cost  of  com¬ 
bat  in  human  life  should  be  minimized, 
for  US  military  personnel,  as  well  as 
enemy  personnel  and  noncombatants. 

•  Minimize  conflict  duration.  Armed  con¬ 
flict  diverts  the  Nation’s  attention  from 
other  issues  and  consumes  finite 
resources  that  could  be  invested  in  other 
areas. 

•  Prevent  repetitive  aggression.  If  a  nation 
has  demonstrated  itself  to  be  an  aggres¬ 


sor  nation,  we  wish  to  preclude  future 
difficulties  with  that  nation  by  destroy¬ 
ing  its  capacity  to  wage  war. 

•  Maximize  force  deployability.  The  loca¬ 
tion  of  the  “next”  conflict  is  unknown 
and  America’s  interests  are  global. 
Because  of  this  it  is  in  U.S.  interests  to 
maximize  its  ability  to  project  combat 
power  to  other  regions  of  the  world  in  a 
minimal  amount  of  time. 

Figure  2  illustrates  how  these  objec¬ 
tives  might  be  organized  and  quantified 
into  a  set  of  fundamental  objectives. 
These  MOE  are  examples  of  the  type  that 
should  be  applied  to  help  provide  deci¬ 
sion  makers  insight  into  many  top-level 
decision  situations.  Judgments  involving 
trades  among  these  types  of  objectives  are 
necessarily  and  appropriately  subjective, 
and  reflect  the  values  (preferences  based 
on  experience)  of  senior  military  leaders. 
These  objectives  identify  explicitly  the 
point  at  which  the  analytical  world  inter¬ 
sects  the  decision  makers’  world.  This 
intersection  should  be  understandable  for 
decision  makers  and  span  the  multiple 
objectives  that  are  affected  by  the  deci¬ 
sion  and  relevant  uncertainties.  The 
MOE  are  not  new;  they  are  a  stable, 
enduring  set  of  measures  that  link  the 
analytic  evaluation  of  policy  and  cam¬ 
paign  level  decisions  to  strategic  objec¬ 
tives. 

(See  REVOLUTIONS,  p.  28) 
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Abstract 

he  use  of  Modeling  and  Simulation 
(M&S)  within  the  Department  of 
Defense  (DoD)  has  undergone  a 
major  change  over  the  last  several  years. 
Unfortunately,  the  vast  majority  of  the 
tools  have  not  kept  up  with  these  changes. 
Current  simulation  systems  do  not  provide 
the  Joint  Warfighter  the  capability  to  con¬ 
duct  joint  and  service  training  across  the 
full  range  of  mihtary  operations  in  a  cost- 
effective  manner.  To  address  this  critical 
shortage,  DoD  has  undertaken  the  develop¬ 
ment  of  the  Joint  Simulation  System 
(JSEMS).  The  JSIMS  Program  was  created 
to  address  the  shortcomings  of  existing 
systems,  implement  a  collaborative  devel¬ 
opment  strategy  for  Joint  and  Service-spe¬ 
cific  capabilities,  reduce  the  resource  over¬ 
head  required  to  conduct  training  exercises, 
and  incorporate  the  common  compliance 
paradigm  for  applicable  standards  and 
guidelines.  This  article  covers  the  rationale 
for  JSIMS  and  approach  involved  in  bring 
this  ambitious  program  to  fruition. 

Warfare  Is  Evolving 

Military  operations  in  the  twentieth  cen¬ 
tury  have  gone  through  three  phases  of 
evolution.  The  first  of  these  started  with 
World  War  II,  the  Attrition  Warfare  era. 
Massed  armies  fighting  discrete  battles 
where  the  side  left  standing  was  declared 
the  victor  characterized  this  phase.  In  the 
late  1970’s  and  early  1980’s  a  new  doctrine 
of  Maneuver  Warfare  came  into  existence. 
Partially  based  on  the  lessons  learned  in 
Viet  Nam,  the  determining  factor  was  the 
ability  to  mass  the  forces  at  the  critical  time 
and  place  to  defeat  the  opponent.  While 


this  phase  is  still  with  us,  we  are  entering  a 
period  where  the  military  has  a  full  spec¬ 
trum  of  missions.  While  the  United  States 
military  has  always  been  used  in  non-tradi- 
tional  roles,  it  is  now  used  in  a  much  broad¬ 
er  range.  New  missions,  such  as  peace 
keeping,  humanitarian  assistance,  and  disas¬ 
ter  relief,  are  stretching  the  envelope  of  the 
training  of  the  armed  services. 

While  the  missions  and  the  methods  of 
the  military  have  changed  drastically,  the 
simulations  to  support  them  have  funda¬ 
mentally  remained  the  same.  For  the  most 
part,  they  are  large,  monolithic,  cumber¬ 
some,  based  on  simplistic  attrition  algo¬ 
rithms,  and  biased  to  a  particular  service. 
As  we  progress  in  the  highly  dynamic 
world  of  the  twenty-first  century  joint  oper¬ 
ations,  the  simulations  used  by  the  military 
must  be  able  to  reflect  the  new  and  evolv¬ 
ing  missions  and  tasks.  To  accomplish  this, 
the  Department  of  Defense  is  undertaking 
the  development  of  a  suite  of  new  models. 
One  of  which,  JSIMS,  is  the  focus  of  this 
article. 

Distributed  Users  And  Developers 

While  the  recent  draw  downs  and  base 
closings  have  reduced  the  number  of  loca¬ 
tions  where  the  military  is  based,  they  still 
tend  to  be  based  by  branch  of  service.  This 
complicates  and  reduces  the  services’  abili¬ 
ty  to  train  together.  To  do  so,  they  have  had 
to  travel  to  central  locations  capable  of 
hosting  the  exercise.  Very  often  this  has 
been  the  driving  cost  of  an  exercise.  Like¬ 
wise,  due  to  teaming,  skiU  mix,  and  politi¬ 
cal  and  economic  considerations,  the 
JSIMS  Enterprise  developers  are  no  longer 
co-located.  Developers  located  in  Massa¬ 
chusetts,  Florida,  Virginia,  California, 
Alabama,  and  the  DC  area  routinely  get 
together  via  video  and  tele-conferences  to 
go  over  development  artifacts  that  have 
been  distributed  electronically.  While  this 
ability  to  establish  a  virtual  presence  has 
greatly  reduced  the  need  for  travel,  much 
work  still  needs  to  be  done  to  refine  and 
understand  the  new  paradigm. 

Warfighter  Vision 

The  Warfighter  vision  is  that  JSIMS  is  a 
simulation  system  that  supports  the  twenty- 


first  century  warfighter’s  preparation  for 
real-world  contingencies.  The  system  pro¬ 
vides  garrison  and  deployed  exercise  capa¬ 
bility  to  meet  current  and  emerging  training 
and  operational  requirements  in  a  timely 
and  efficient  manner.  By  interfacing  to  the 
warfighter’s  real  go-to-war  systems,  the 
view  into  the  simulation  world  mirrors  that 
of  the  real  world. 

JSIMS  will  support  Unified  Combat 
Commands,  Services  and  Joint  Task  Force 
training  in  all  phases  of  militaiy  operations. 
It  will  also  support  specific  Universal  Joint 
Task  List  (UJTL)  and  service  task  lists 
requirements  for  strategic-theater,  opera¬ 
tional  and  tactical  tasks. 

Consolidated  Use  Cases 

As  part  of  its  definitional  phase,  seven 
prototypical  use  cases  were  developed. 
Each  one  of  these  represents  a  notional 
space  in  terms  of  fidelity  and  level  of  repre¬ 
sentation  in  which  a  class  of  user  will  use  a 
JSIMS  composition.  The  first  two  use 
cases,  CINC/JTF/Component/Agency 
Training  and  Service/Agency  Training, 
cover  the  same  space  as  the  current  Joint 
Training  Confederation  (JTC).  Together, 
these  represent  the  Initial  Operational 
Capability  (IOC)  functionality  of  JSIMS. 

As  JSIMS  evolves  to  Full  Operational 
Capability,  the  remaining  five  use  cases. 
Planning  Analysis,  Crew/Team  Rehearsal, 
Other  Military  Education,  Senior  Officer 
Education,  and  Doctrine  Development  will 
be  met.  At  this  time,  JSIMS  will  have 
effectively  covered  almost  the  entire  spec¬ 
trum  of  the  model  space. 

JSIMS  will  not  be  used  in  isolation. 
Rather,  it  is  being  designed  to  provide  an 
overall  context  to  the  synthetic  battlespace. 
It  does  this  by  interacting  with  other  sys¬ 
tems  via  well-defined  interfaces.  Per  cur¬ 
rent  DoD  policy,  the  primary  means  of 
interacting  with  these  systems  will  be  via 
the  High  Level  Architecture  (HLA)  data 
interchange  mechanisms. 

Technical  Vision 

The  Technical  vision  is  that  JSIMS  is  a 
single,  distributed,  seamlessly  integrated 
simulation  environment.  It  includes  a  core 
OVERVIEW,/?.  72) 
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OVERVIEW 

(continued  from  p.  11) 

infrastructure  and  mission  space  objects, 
both  maintained  in  a  common  repository. 
A  common  simulation  engine  includes  sys¬ 
tem  software  to  run  on  commercially  avail¬ 
able,  open  architecture  hardware  and  net¬ 
works.  These  can  be  composed  to  create  a 
simulation  capability  to  support  joint  or  ser¬ 
vice  training,  rehearsal,  or  education  objec¬ 
tives. 

Development  Plan 

The  JSIMS  developmental  schedule  is 
broken  down  into  three  major  phases.  The 
first  phase  is  the  program  definition,  which 
came  to  an  end  in  the  April,  1998  time- 
frame.  During  this  timeframe,  the  contracts 
for  the  seven  development  efforts  (shown 
in  Table  1)  were  awarded^  the  JSIMS 
Enterprise  was  stood  up,  the  Acquisition 
Program  Baseline  (APB)  was  signed,  and 
Build  0  was  completed.  The  signing  of  the 
APB  represented  a  major  advancement  in 
the  way  programs  are  conducted.  For  the 
first  time  that  anybody  could  remember, 
twenty-nine  signatories,  from  the  Program 
Managers  to  the  service  Acquisition  Execu¬ 
tives  and  Operations  Deputies,  all  signed 
up  to  a  common  approach,  set  of  require¬ 
ments,  and  schedule.  A  by-product  of  the 
APB  was  the  IOC  requirements  walk¬ 
through.  This  allowed  all  of  the  developers 
to  sit  down  with  the  requirements  providers 
to  gain  a  better  understanding  of  the  func¬ 
tionally  needed  to  meet  the  user’s  needs. 
The  importance  of  Build  0  was  that,  while  a 
process  build,  it  showed  that  the  Enterprise 
could  use  the  JSIMS  Object  Oriented 
Process  (JOOP)  to  build  executing  soft¬ 
ware. 

The  next  two  components  are  both  data¬ 
bases,  but  they  serve  different  purposes. 
The  Common  Data  Infrastructure  (CDI) 
deals  with  the  runtime  data  generated  in  an 
exercise.  These  data  are  used  to  restart  the 
system  and  to  evaluate  the  exercise.  The 
Common  Database  is  used  to  store  the  stat¬ 
ic  data  that  are  needed  to  help  compose  the 
simulation  for  the  exercise  and  generate  the 
scenario.  The  exercise  generation  and  eval¬ 
uation  tools  are  parts  of  the  next  compo¬ 
nent,  the  Life  Cycle  Applications.  The 
JSIMS  LCA  suite  is  an  integrated  set  of 
tools  to  aid  the  users  in  the  construction, 
execution,  and  analysis  of  an  exercise. 
These  are  needed  to  help  hide  the  complex¬ 
ity  and  details  of  the  implementation  of  the 


Program 

Development  Domain 

Joint  Simulation  System  (JSIMS) 

Joint  Program  Office  (JPO) 

Joint  Domain  and  Core  Infrastructure 

National  Air  and  Space  [Warfare] 

Model  (NASM) 

Air  and  Space  Domain 

JSIMS  Maritime 

Maritime  Domain 

Warfighter  Simulation  2000  (WARSIM) 

Land  Domain 

WARSIM  Intelligence  Module  (WIM) 

Tactical  Intelligence 

National  Simulation  (NATSIM) 

Overhead  and  National  Intelligence 

Joint  Signal  Simulation  (JSIGSIM) 

Signal  Intelligence 

Table  1.  The  seven  programs  making  up  the  JSIMS  Enterprise 


Other  Enclaves 


_ UL, 

Life  Cycle 
Applications 

Common 

Services 

DBMS  Tools 
and  Services 
(Browsers. 

J ^ 

Servers,  Mgmt 
tools,  etc.) 

‘  . ^ 

u4I  Surrogate 

i, — 

Lines,  Tactical 
Comms,  IPC,  «tc. 


— L 

Non-GCCS 

GCCS 

Operational  C4I 

Operational  C4I 

Systems 

Systems 

Figure  1.  The  JSIMS  Components 


users.  The  final  component  is  the  Enclave 
Ingress  /  Egress  module  or  Security  Gate¬ 
way.  The  gateway  acts  as  a  trusted  guard 
allowing  informationto  flow  between 
JSIMS  enclaves  operating  at  different  secu¬ 
rity  classifications  or  compartments. 

System  Development  Process 

Traditional  software  development 
process  has  used  the  big  bang  waterfall 
development  process.  Each  step  logically 
follows  the  other  with  the  products  coming 
down  the  waterfall.  At  the  end  of  the  water 
the  system  is  delivered  to  the  user.  Experi¬ 
ence  has  shown  that  this  is  not  an  effective 
way  of  building  systems  that  are  not  fully 
understood  at  the  outset  or  have  the  require¬ 
ments  change  during  development.  These 


types  of  systems,  which  include  virtually  all 
simulation  systems,  lend  themselves  to  an 
iterative  developmental  approach.  Each 
iteration  is  divided  into  roughly  two  phases. 
The  first  of  these  is  the  specification  of  a 
conceptual  model  based  upon  the  current 
and  best  understood  set  of  requirements. 
This  model  represents  a  formalism  of  the 
real  world.  Since  the  entire  world  will  not 
be  modeled,  the  system  requirements  are 
used  to  help  filter  and  guide  what  is 
included. 

Once  the  conceptual  model  is  built,  the 
requirements  are  again  used  to  determine 
what  functionality  should  be  implemented 
in  software  vice  met  by  manual  or  external 
processes.  The  software  system  is  then, 

(See  OVERVIEW,/?.  25) 
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David  E.  Kendrick 

Veridian,  Veda 
Operations  Devel¬ 
opment  Group 


Mike  Hopkins 

Defense  Modeling 
and  Simulation 
Office 


Have  you  ever  had  the  dilemma  of 
having  to  find  a  credible  source  of 
knowledge  to  support  a  model 
development  or  to  populate  a  model  data¬ 
base  for  a  specific  application?  Those  of 
you  who  have  know  the  time  expended  in 
these  type  efforts  is  substantial.  The 
Defense  Modeling  and  Simulation  Office 
sponsored  data  source  cataloging  effort  has 
provided  the  Department  of  Defense  (DoD) 
Modeling  and  Simulation  (M&S)  Commu¬ 
nity  with  a  knowledge  acquisition  (KA) 
resource  that  expedites  these  efforts  across 
the  spectrum  of  M&S  use. 

The  credibility  of  models  and  simula¬ 
tions  is  directly  correlated  to  their  represen¬ 
tation  of  real  tasks,  systems  and  environ¬ 
ments.  Locating  and  acquiring  the  best 
available  knowledge  (data/information)  is  a 
key  step  in  building  that  credibility.  To 
address  this  need,  DMSO,  in  accordance 
with  direction  within  DoD  Directive 
5000.59-P  (DoD  Modeling  and  Simulation 
(M&S)  Master  Plan)  to  strengthen  the  use 


of  M&S  in  the  three  functional  areas;  train¬ 
ing,  analysis  and  acquisition,  established  the 
Authoritative  Data  Source  (ADS)  Working 
Group  (WG)  in  February  1994.  The 
ADSWG  consists  of  Service  and  Compo¬ 
nent  modeling  and  simulation  office  repre¬ 
sentatives  and  M&S  Executive  Agent 
(M&S  EA)  representatives. 

The  ADS  project  is  intended  to  improve 
data: 

•  Credibility  (by  designating  authoritative 
data); 

•  Composability  (by  describing  standard 
data);  and, 

•  Cost  (by  rapid  on-line  identification  and 
selected  access). 

The  ADS  Working  Group  has  developed 
three  key  products: 

•  The  ADS  designation  process,  (Compo¬ 
nent  sources  are  labeled  with  levels  of 
authority); 

•  The  ADS  taxonomy  of  data  source  cate¬ 
gories  (taxonomy  with  definitions  facili¬ 
tate  searches);  and, 

•  The  ADS  library  (list  of  sources  with 
metadata  descriptions). 

There  are  four  steps  to  the  process  that 
provides  designated  data  sources  to  the 
M&S  community. 

1.  Identify  the  sources.  Senior  representa¬ 
tives  at  the  Service  and  Component 
modeling  and  simulation  offices  were 
requested  to  provide  prioritized  listings 
of  their  sources.  The  search  was 
expanded  by  (1)  obtaining  inputs  from 
major  M&S  users,  (2)  asking  identified 
points  of  contact  in  key  M&S  organiza¬ 
tions  to  identify  other  sources  and  (3) 
search  of  US  Government  and  Military 
web  sites. 

2.  Gather  a  standard  set  of  information. 

The  standard  set  of  metadata  being  gath¬ 
ered  was  developed  and  approved  by  the 
ADSWG.  The  focus  is  on  providing  the 
information  required  during  the  knowl¬ 
edge  acquisition  phase  of  typical  model 
development  or  employment.  The  con¬ 


tent  of  the  ADS  metadata  has  been 
cross-walked  with  International  Stan¬ 
dards  Organization  standards,  the  Feder¬ 
al  Geographic  Data  Committee  stan¬ 
dards  for  metadata  and  the  Defense  Data 
Dictionary  System  (DDDS).  The  stan¬ 
dard  set  of  metadata  being  included  in 
the  Library  has  recently  been  expanded 
to  include  detailed  data  quality  informa¬ 
tion  identified  as  required  by  the  DoD 
VV&A  working  group’s  Data  Quality 
Tiger  Team  metadata  template. 

3.  Designate  each  source  with  a  level  of 
Authority.  Designation  provides  M&S 
users  with  key  information  necessary  to 
determine  the  appropriateness  of  the  data 
for  the  need.  Designations  are  assigned 
by  the  appropriate  Service  or  Compo¬ 
nent  responsible  for  the  creation  of  the 
information. 

Three  levels  of  authority  have  been 
assigned  in  the  past.  Those  levels  were; 
Authoritative,  Approved  and  Other.  The 
designation  naming  convention  and 
associated  definitions  are  being  modi¬ 
fied.  As  of  the  writing  of  this  article  the 
new  names  and  definitions  have  been 
formally  approved  by  the  Army,  Air 
Force  and  Air  &  Space  Executive  Agent. 

The  new  definitions  are: 

Category  I  —  A  recognized  Service  or 
National  data  production  center  with 
applicable  mission  statement,  industry 
provider,  or  source/product  that  has 
established  and  documented  production 
quality  control  procedures  and  quality 
controls  the  data  produced.  These  orga¬ 
nizations  or  sources/products  have  a  reli¬ 
able  performance  history.  They  have 
well-defined  data  metrics  and  significant 
metadata  information,  sufficient  to  satis¬ 
fy  all  priority  one  Data  Quality  Template 
requirements  (ADS  mandatory  metadata 
fields),  available  according  to  a  recog¬ 
nized  metadata  standard. 

Category  11  —  A  data  producer  or 
source/product  designated  by  a  Compo- 

(See  DATA  SOURCES,/?.  14) 
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Figure  1.  ADS  Process 


Figure  2.  Top  Level  Taxonomy 


DATA  SOURCES 

(continued  from  p.  13) 

nent.  Includes  data  providers  and 
sources/products  that,  while  not  of  the 
Category  I  stature,  have  become  de  facto 
providers  of  data  or  have  unique,  one-of- 
a-kind  data  sets  and  employ  quality  con¬ 
trol  procedures.  This  category  includes 
providers  of  individual  data  sets  that 
have  been  accepted  or  designated  as 
research  grade  data  sets.  The  category 
may  also  include  industry  providers  who 
establish  Component-approved  M&S 
support  systems.  Their  data  may  or  may 
not  have  well  defined  data  metrics  and 
their  metadata  only  partially  satisfy  the 
Data  Quality  Template  priority  one 
requirements  (ADS  mandatory  metadata 
fields). 

Category  III  —  A  producer,  source,  or 
product  that  is  not  Categoiy  I  or  Catego¬ 
ry  n  but  is  available  for  use  as  deemed 
appropriate  by  the  user. 

T  suffix  -  Indicates  source  had  been 
previously  designated  at  the  comparable 
level  in  accordance  with  the  previous 
designation  categories  and  definitions. 
However,  the  metadata  set  requires  com¬ 
pletion  of  applicable  new  data  quality 
elements  of  information  and  subsequent 
review  by  the  designating  authority  to 
ensure  the  level  is  still  appropriate.  It  is 
envisioned  that  this  will  occur  with  the 
routine  update/  review  schedule  of  each 
source. 

The  old  definitions  were: 

Authoritative  Data  Source:  A  source  or 
product,  which  has  undergone  producer 
data  W&C  activities. 

Approved  Data  Source:  A  source  or 
product  that  has  been  generated  for  a 
specific  purpose  and  has  been  reviewed 
by  the  proponent  and  user  and  they  agree 
to  its  validity  and  appropriateness  for 
that  use. 

Other  Data  Source:  A  source  or  prod¬ 
uct  that  has  not  been  designated  Authori¬ 
tative  or  Approved  but  is  available  for 
use  as  appropriate. 

4,  Make  the  information  readily  avail¬ 
able  to  the  M&S  community.  The 

value  of  the  ADS  metadata  and  designa¬ 
tion  lies  in  its  use  to  reduce  knowledge 
acquisition  time.  The  metadata  and 
associated  designations  that  have  been 


collected  in  the  ADS  Library  are  avail¬ 
able  on-line  as  one  of  the  major 
resources  of  the  Modeling  and  Simula¬ 
tion  Resource  Repository  (MSRR)  at 
http://ads.msrr.dmso.mil. 

A  graphical  depiction  of  the  process  is 
provided  in  Figure  1.  Initial  direction  for 
the  project  was  to  attack  the  task  from  the 
top  down,  requesting  the  senior  Model  and 
Simulation  representative  of  the  Services, 
Agencies  and  Executive  Agents  (EAs)  to 
provide  a  list  of  prospective  sources  to 
DMSO.  DMSO  requested  that  the  lists  be 


prioritized  to  focus  early  metadata  collec¬ 
tion  efforts  on  those  sources  that  were  most 
likely  authoritative  in  nature.  In  FY97  a  sec¬ 
ond  direction  was  added.  DMSO  requested 
major  M&S  end-users  within  the  communi¬ 
ty  to  identify  the  sources  they  were  current¬ 
ly  using  and  requirements  they  had  that 
were  not  being  adequately  met,  a  bottom  up 
approach.  This  bottom-up  approach  also 
provides  the  potential  for  early  rewards 
from  the  project  by  identifying  sources  that 
would  meet  current  shortfalls.  Both  top- 
down  and  bottom-up  approaches  have  iden¬ 
tified  a  large  number  of  sources.  More 
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Service/Agency 

Designated 

SME/Orgs 

In  Process 

Sources  Identified 

Army 

152 

17/09 

79 

413 

Air  Force 

23 

00/01 

55 

196 

Navy 

3 

00/02 

68 

36 

Marine 

0 

00/00 

17 

10 

DIA 

4 

00/00 

1 

6 

NIMA/Terrain  EA 

26 

00/00 

27 

0 

Air  &  Space  EA 

0 

00/00 

24 

0 

Ocean  EA 

0 

00/01 

46 

40 

OSD 

142 

00/03 

152 

31 

JCS 

110 

00/00 

41 

24 

Other  Gov’t 

6 

00/00 

31 

36 

Commercial 

13 

00/00 

7 

0 

Foreign 

1 

00/00 

_0 

1 

TOTAL 

480 

17/16 

548 

793 

Figure  3.  Detailed  breakout  of  Sources  by  Agency 


recently,  DMSO  has  been  working  directly 
with  key  M&S  developers  to  identify  data 
sources  required  to  support  their  knowledge 
acquisition  projects.  These  approaches  are 
supplemented  by  independent  research  by 
the  contractor  team  performing  the  task. 

Sources  identified  are  provided  to  Vend- 
ian’s,  Applied  Technology  Group  (contract¬ 
ed  by  DMSO  beginning  in  1996  to  support 
the  Services,  Components  and  EAs).  Each 
source  is  contacted  to  initiate  the  metadata 
collection  effort,  if  it  is  determined  to  be 
beneficial,  on-site  briefings  are  conducted 
to  more  clearly  explain  the  project.  The 
standard  set  of  information  (metadata)  is 
collected  on  each  source. 

In  addition  to  gathering  the  required  set 
of  metadata,  responsibilities,  definitions  of 
data  sources  and  data  centers  and  develop¬ 
ment  of  guidelines  for  verification  and  vali¬ 
dation  (V&V)  and  how  they  pertain  to 
M&S  data  are  discussed  with  the  sources. 
Veridian  compiles  the  resulting  information 
into  a  report,  updates  the  ADS  Library  and 
provides  the  information  to  DMSO  at  six- 
month  intervals. 

The  ADS  Working  Group  developed  a 
taxonomy  of  data  source  categories  (13  top- 
level  categories,  see  Figure  2,  and  373  sub¬ 
categories  not  shown)  to  organize  the  iden¬ 
tification  of  sources,  production  of  metadata 
and  to  assist  ADS  Library  search  and  dis¬ 
covery.  Formal  definitions  have  been 
developed  and  coordinated  for  each  catego¬ 
ry.  The  ADS  taxonomy  “organizes”  M&S 
information  according  to  the  natural  divi¬ 
sions  of  labor  among  data  producers. 

At  present,  the  ADS  Library  is: 

•  Centrally  managed  by  DMSO; 

•  A  single,  homogenous  data  set  stored  in 
one  location;  and, 

•  Accessible  via  the  Modeling  and  Simula¬ 
tion  Resource  Repository. 

With  the  maturing  of  the  Modeling  and 
Simulation  Resource  Repository  (MSRR) 
the  ADS  Library  will  become  decentralized 
but  transparently  linked  at  various  Compo¬ 
nent  and  Agency  sites.  Agreement  on 
metadata  constructs,  content  and  common 
search  processes  are  making  this  possible. 

Designations  are  determined  through  the 
following  process.  DMSO  reviews  the 
information  collected  by  Veridian  for  com¬ 
pleteness.  Sources  are  sorted  by  Compo¬ 
nent.  The  metadata  sets  are  then  provided 
to  the  appropriate  Component.  Each  Com¬ 
ponent  then  designates  each  source  accord¬ 


ing  to  Component-specific  procedures. 
Designations  and  associated  metadata  are 
returned  to  DMSO  for  inclusion  into  the 
ADS  Library.  Sources  that  are  not  clearly 
identifiable  as  belonging  to  a  specific  Com¬ 
ponent,  are  non-DoD,  or  commercial 
sources  are  sent  to  all  Components  with 
interest  in  the  respective  data  area.  If  differ¬ 
ent  levels  of  authority  are  recommended  by 
different  Components,  the  ADSWG 
resolves  the  difference  and  assigns  the 
authority  to  be  placed  on  the  source  in  the 
hbrary,  appropriate  comments  are  included 
in  the  source  description. 

The  product  of  the  ADS  Project  can  be 
effectively  utilized  to  reduce  the  knowledge 
acquisition  phase  during  both  model  devel¬ 
opment  and  employment.  As  an  example:  a 
Conceptual  Model  of  the  Mission  Space 
(CMMS)  developer  interested  in  capturing 
the  necessary  aspects  of  the  Army’s 
involvement  in  logistics  over  the  shore  can 
use  the  ADS  tool  to  locate  and  access  the 
doctrinal  sources  both  Joint  and  Service  to 
understand  the  concepts.  He  would  enter 
the  ADS  database  and  search  for  logistics, 
logistics  over  the  shore,  or  more  specific 
key  words.  The  search  would  return  all 
data  sources  that  are  specifically  related  to 
the  key  words,  showing  the  level  of  authori¬ 
ty  associated  with  each.  The  list  would 
contain  documents,  subject  mater  experts 
and  instance  data  sources.  The  CMMS 
developer  would  likely  be  most  interested 
in  the  doctrine.  He  would  click  on  one  of 
the  sources  listed,  as  an  example.  Joint  Pub 
4-01.6  Joint  Tactics  Techniques,  and  Proce¬ 
dures  for  Joint  Logistics  over  the  Shore 


(JLOTS),  and  would  be  provided  the  meta¬ 
data  on  that  specific  source.  If  that  metada¬ 
ta  tells  the  developer  that  this  source  con¬ 
tains  pertinent  information  to  his  task  he  can 
click  on  the  hypertext  link  (Note:  Not  all 
sources  listed  in  the  ADS  library  have 
hypertext  links  that  will  take  you  to  the  end 
data.  Release  of  the  data  is  still  controlled 
by  the  data  producer  or  repository  and 
many  require  an  approval  process)  and  be 
taken  to  the  Joint  Electronic  Library  where 
he  can  read  the  actual  publication  and 
acquire  a  printed  copy  if  necessary.  After 
reading  the  doctrine  the  developer  could 
return  to  the  source  metadata  and  if  neces- 
saiy  contact  the  publication  POC  for  further 
detail.  He  would  then  go  back  to  the  search 
results  looking  for  Army  doctrinal  docu¬ 
ments  or  to  identify  Subject  Matter  Experts 
to  provide  sufficient  detail  to  the  CMMS 
abstraction. 

The  ADS  Library  is  availaUe  on-line 
today  and  contains  1061  sources.  The  des¬ 
ignation  process  for  the  last  library  update  is 
still  in  progress.  Today  the  library  contains 
287  Authoritative,  184  Approved,  9  Other, 
17  SMEs,  16  Organizations,  with  548  still 
in  the  designation  progress.  Additional 
sources  are  identified  daily  with  the  current 
list  of  sources  to  be  investigated  for  possible 
inclusion  standing  at  793.  Figure  3  is  a 
detailed  breakout  of  sources  by  service. 

The  current  library  supports  a  very 
robust  key  word  or  category  search  capabil¬ 
ity  and  a  number  of  reports  can  be  obtained 
from  the  database. 

Use  of  models  and  simulations  in  the 
(See  DATA  SOURCES,/?.  16) 
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DATA  SOURCES 

(continued  from  p.  15) 

operational,  acquisition,  testing,  research 
and  development,  and  training  environ¬ 
ments  will  continue  to  expand.  Authorita¬ 
tive  Data  Sources  are  a  key  element  in 
DMSO’s  efforts  to  improve  credibility, 
composability,  and  reduce  the  cost  of 
knowledge  acquisition. 
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15th  ISMOR:  Another  Success  Story 


On  31  August 
1998,  the 
15  th  Interna¬ 
tional  Symposium  on 
Military  Operational 
Research  opened  with 
the  traditional  wel¬ 
coming  activities  in 
the  Roberts  Hall  Mess 
at  the  Royal  Military 
College  of  Science,  Shrivenham,  England. 
The  Symposium  was  chaired  with  his  usual 
skill  by  David  Faddy,  following  in  the 
footsteps  of  the  late  Professor  Ronnie 
Shephard,  founder  of  the  ISMOR  series. 
The  Commandant  of  RMCS,  Major  Gener¬ 
al  Ash  Irwin,  CBE,  gave  the  welcoming 
address  to  the  participants  to  initiate  the  for¬ 
mal  sessions,  running  from  1  through  4 
September,  culminating  in  an  open  forum 
that  last  morning.  The  theme  —  Is  a  revo¬ 
lution  in  analysis  required?  —  was  identi¬ 
fied  in  a  surprisingly  large  number  of  the 
presentations.  Normally,  a  theme  is  not  as 
well  covered  in  professional  meetings  such 
as  this.  No  conclusion  was  reached  as  to  the 
answer  to  the  question  posed  by  the  theme; 
there  were  as  many  negative  responses  as 
positive  and  “maybe”  responses. 

Registration  and  administration  of  the 
Symposium  was  in  the  always  capable 
hands  of  Trish  Follows,  who  has  the  won¬ 
derful  record  of  managing  all  15  of  the 
symposia;  she  was  assisted  by  the  equally 
efficient  Maggie  Floyd. 

A  total  of  30  papers  were  presented. 
Another  tradition  of  the  ISMOR  series  is 
that  no  proceedings  are  published.  The  pre¬ 
senters  bring  sufficient  copies  of  their 
papers  so  that  all  participants  depart  with  a 
complete  set,  eliminating  the  need  to  print 
proceedings  consisting  of  the  collected 
papers.  [Readers  interested  in  a  list  of 
papers  with  authors’  addresses  may  contact 
E.  P,  Visco  at  3752  Capulet  Terrace,  Silver 
Spring,  MD  20906;  voice:  301.598.8048; 
facsimile:  301.438.0395;  e-mail: 
visco03@ibm.net  for  a  copy.]  By  national 
origin,  papers  were  distributed  as:  12  from 
the  United  Kingdom,  10  from  the  United 
States,  three  from  Canada,  two  from 
NATO  (one  from  the  Consultation,  Com¬ 
mand  and  Control  Agency  and  one  from 
the  Allied  Rapid  Response  Corps)  and  one 
each  from  Germany,  the  Netherlands  and 


Sweden.  To  whet  your  appetites,  here  are 
some  of  the  titles,  to  give  the  flavor  of  the 
Symposium:  “When  Systems  are  Simula¬ 
tions:  T&E,  VV&A  or  Both?”  (P.  A. 
Glasow  and  M.  Boroswski,  USA;  present¬ 
ed  by  Ms.  Glasow);  “Stochastic  Analysis  of 
US  Army  Deployments  and  Excursions” 
(ETC  P.  J.  DuBois,  USA);  “Contrary 
Schools  of  Thought  within  Military  Deci¬ 
sion  Making  Groups”  (F.  Cameron,  Cana¬ 
da);  “Project  Albert:  A  Well  Tempered 
Revolution”  (A.  G.  Brandstein,  G.  E. 
Horne  and  Capt  T.  M.  Strycharz,  USA; 
presented  by  Dr.  Brandstein);  “SALOMO, 
Decision  Support  to  Airbase  Logistics”  (N. 
vanElst,  Netherlands);  “Representation  of 
Fear  and  Shock  in  Combat  Models”  (J. 
Moffat  and  L.  Dodd,  UK;  presented  by 
Mr.  Moffat);  “Peacekeeping  in  Bosnia: 


The  theme —/s  a  revolution  in 
analysis  required?— was  identified 
in  a  surprisingly  large  number 
of  the  presentations.  Normally,  a  theme 
is  not  as  well  covered  in  professional 
meetings  such  as  this. 


Fatality  Estimates”  (C.  A.  Lawrence, 
USA);  and  “Complexity  Theory  -  The  Sim¬ 
ple  Answer  to  All  Our  Problems”  (P.  L. 
Grainger,  UK). 

Participation  in  the  Symposium  was  dis¬ 
tributed  as:  United  Kingdom:  58;  United 
States:  25  (including  2  from  USAEU- 
COM);  Canada,  Germany  and  Turkey:  4 
each;  NATO  (2  from  the  ARRC  and  1  from 
C-cubed),  the  Netherlands  and  Sweden:  3 
each;  France  and  Israel:  2  each;  and  Den¬ 
mark  and  Singapore:  1  each.  [The  numbers 
may  not  be  exact  since  they  are  taken  from 
the  list  of  participants  prepared  from 
advance  registration  and  do  not  reflect  last 
minute  changes.] 

MORS  was  well  represented,  although 
long-time  ISMOR  participants  Richard  1. 
WUes,  Executive  Vice-President,  and  Pro¬ 


fessor  Peter  Purdue  were  not  able  to 
attend  15  ISMOR  because  of  other 
demands.  The  MORSians  at  15  ISMOR 
included  Dr.  George  Akst,  past  USMC 
Sponsor’s  Representative;  MAJ  B.  J.  Bar¬ 
ns;  W.  J.  Bauman;  Mary  F.  Bonnet;  Dr. 
Alfred  G.  Brandstein,  Director;  Dr.  Paul 
Deitz;  LTC  Patrick  J.  DuBois;  LCDR  Aas- 
geir  Gangsaas;  Priscilla  A.  Glasow,  Direc¬ 
tor;  Dr.  Robert  L.  Helmbold;  Dr.  Jacque¬ 
line  Henningsen,  FS;  Frank  Mahncke; 
LTC  John  B.  Musser;  Professor  David  A. 
Schrady,  FS  and  Past  President;  Eugene  P, 
Visco,  FS;  and  Dr.  Daniel  Willard,  who 
represented  Walter  W.  Hollis,  FS,  Army 
Sponsor  of  MORS  and  a  long-time  sup¬ 
porter  of  ISMOR. 

The  Professor  Ronnie  Shephard  Memo¬ 
rial  Lecture  was  delivered,  as  usual,  at  the 
banquet  on  3  September.  In  a  modest  break 
with  tradition,  the  talk  was  presented  by  a 
military  man,  LTG  E.  F.  G.  Burton,  OBE, 
MA,  Deputy  Chief  of  the  Defence  Staff 
(Systems),  UK.  General  Burton  is  well- 
known  to  ISMOR  veterans  (as  a  friend  and 
associate  of  Professor  Ronnie  and  dedicat¬ 
ed  supporter  of  the  ISMOR  series  at 
RMCS,  particularly  notable  during  his 
tenure  as  Commandant)  and  to  members  of 
the  US  defense  community  (from  his  days 
as  attache  in  the  British  Embassy  and  in 
subsequent  assignments  on  the  Ministry  of 
Defence  staff).  In  a  departure  from  the 
reminiscent  nature  of  earlier  talks,  particu¬ 
larly  by  operations  analysts  of  the  World 
War  n  and  Korean  War  periods.  General 
Burton  made  a  series  of  challenges  to  the 
military  operations  analysis  community. 
General  Burton  placed  emphasis  on  mat¬ 
ters  such  as  improved  understanding  of 
human  behavior  in  and  contributions  to 
military  operations  and  trade-off  analyses 
to  assist  in  the  higher  level  decision  mak¬ 
ing  in  the  ministries  and  departments  of 
defense. 

Plans  are  underway  for  16  ISMOR  to 
be  held  at  the  RMCS  about  6  September 
1999.  Suggestions  for  a  theme  or  other 
suggestions  are  welcomed.  Ideas  can  be 
sent  to  Mr.  Visco,  address  above,  or  David 
Faddy,  CDE  Lanchester  Building  A3, 
DERA,  Ively  Road,  Farnborough, 
Hantshire  GUI 4  OLX,  United  Kingdom; 
voice:  011-44-1252-396200;  facsimile: 
011-44-1252-396207.0 
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The  United  States  Military  Academy  at  West  Point 

by  LTC  Steven  Horton 

e-mail:  AS9492@exmaiLusma.army.mil 


The  67th  Military  Operations  Research 
Society  Symposium  (MORSS)  will 
be  held  at  the  United  States  Military 
Academy  22-24  June  1999.  Since  it  was 
founded  on  16  March  1802,  USMA  and  its 
graduates  have  played  an  important  role  in 
the  history  of  the  United  States.  Its  mission 
is  to  educate,  train  and  inspire  the  Corps  of 
Cadets  so  that  each  graduate  is  a  commis¬ 
sioned  leader  of  character  who  is  committed 
to  Duty,  Honor,  Country;  a  career  in  the 
United  States  Army;  and  a  lifetime  of  ser¬ 
vice  to  the  nation.  Graduates  of  the  Acade¬ 
my  include  Dwight  D.  Eisenhower,  Dou¬ 
glas  MacArthur,  George  S.  Patton,  H. 
Norman  Schwarzkopf,  John  J.  Pershing, 
Robert  E.  Lee,  Ulysses  S.  Grant,  and  astro¬ 
nauts  Frank  Borman,  Mike  Collins  and 
Edwin  Aldrin. 

This  summer,  you  have  the  opportunity 
to  visit  West  Point  and  become  a  part  of 
what  has  been  accomplished  here  by  attend¬ 
ing  the  67th  MORSS.  This  article  was  writ¬ 
ten  to  motivate  you  to  do  so!  First,  to  inspire 
you,  we  provide  a  few  historical  notes  about 
mathematics  and  operations  research  at 
USMA.  Next  we  offer  some  information 
about  things  to  do  when  you  come  to  West 
Point.  Finally,  we  give  some  resources  you 
can  use  to  find  additional  information  about 
West  Point  and  the  67th  MORSS. 

Mathematics  and  Operations 
Research  at  West  Point 

USMA  graduates  have  made  significant 
contributions  in  virtually  every  field  of 
human  endeavor,  including  the  relatively 
new  field  of  operations  research  and  its  par¬ 
ent  discipline  of  mathematics.  Mathematics 
has  been  an  important  part  of  the  USMA 
curriculum  from  the  earliest  days  of  the 
institution.  The  act  of  Congress  establishing 
USMA  in  1802  provided  for  an  engineer,  in 
the  rank  of  major,  to  be  Superintendent,  and 
two  assistant  engineers,  in  the  rank  of  cap¬ 
tain,  to  serve  as  educators  of  the  Academy. 
Captain  Jared  Mansfield  was  one  of  the 
assistant  engineers  and  was  appointed  Act¬ 
ing  Professor  of  Mathematics  in  May  1802, 


so  the  Academy’s  first  educator  was  a  math 
teacher.  Both  Robert  E.  Lee  and  Omar 
Bradley  taught  mathematics  at  West  Point 
as  well.  Omar  Bradley  and  Dwight  D. 
Eisenhower  served  together  in  the  1930’s 
under  then  Chief  of  Staff  of  the  Army  Dou¬ 
glas  MacArthur  solving  problems  of  mobi¬ 
lization,  mechanization  and  the  relationship 
of  military  power  to  industrial  capacity.  Of 
course  today  we  would  call  much  of  this 
work  operations  research,  but  that  term  had 
not  yet  been  introduced.  Since  World  War 
II,  operations  research  has  become  an 
important  part  of  the  curriculum  at  USMA 
as  it  has  at  other  institutions.  Today  the 
Department  of  Mathematical  Sciences  and 
the  Department  of  Systems  Engineering  co¬ 
sponsor  the  operations  research  program. 
USMA,  with  leadership  from  these  two 
departments,  will  carry  on  these  educational 
traditions  and  simultaneously  look  forward 


to  the  Academy’s  third  century. 

Things  to  do  in  the  West  Point  Area 

West  Point  and  the  surrounding  area 
offers  visitors  a  great  variety  of  things  to  do. 
Many  of  the  most  popular  attractions  at  West 
Point  are  within  easy  walking  distance  of  the 
Symposium  site,  Thayer  Hall,  Trophy  point, 
the  cadet  barracks  and  the  cadet  chapel  are 
all  less  than  a  mile  away.  These  sites  are 
visited  by  thousands  of  people  from  through¬ 
out  the  world  every  year.  For  those  with  an 
interest  in  physical  fitness  and  projectile 
motion  problems,  the  West  Point  golf  course 
is  close  by  and  open  to  the  public.  In  addi¬ 
tion,  tennis  courts  are  available  a  short  walk 
from  Thayer  Hall.  The  Cadet  bookstore  is  in 
Thayer  Hall  and  will  be  open  every  day  dur¬ 
ing  the  Symposium.  There  are  gift  shops  in 
both  the  West  Point  Visitors’  Center  and  the 
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Mr.  Brian  Engler,  Systems  Planning  and  Analysis 
e-mail  BENGLER@SPA.com 


new  Alumni  Center. 

The  Hudson  Valley  has  many  other 
attractions.  For  those  with  an  interest  in  his¬ 
tory,  the  last  encampment  of  George  Wash¬ 
ington' s  Continental  Army  is  near  New¬ 
burgh,  less  than  a  half  an  hour  from  West 
Point  by  car.  Franklin  D.  Roosevelt’s 
home  and  museum  is  about  an  hour  away 
across  the  Hudson  in  Hyde  Park,  NY.  Also 
near  Hyde  Park  is  the  celebrated  Culinary 
Institute  of  America  which  offers  meals  pre¬ 
pared  by  some  of  the  finest  chefs  de  cuisine 
in  the  world.  If  shopping  is  your  desire,  the 
Woodbury  Commons  outlet  mall  is  about  20 
minutes  away  in  Harriman,  NY,  A  more 
quaint  shopping  experience  can  be  found  in 
the  old  village  of  Cold  Spring  just  across  the 
river  from  West  Point.  Finally,  New  York 
City  is  only  about  an  hour  away  by  car  or 
train. 


I  ‘‘M)nnatioii  about  MORS  and  the 
67lh  MORSS  is  available  at 
http://www.mors.org.” 

For  Further  Information 

More  information  on  many  of  the  topics 
discussed  in  this  article  is  available  on  the 
World  Wide  Web.  Information  about 
MORS  and  the  67th  MORSS  is  available  at 
http://www.mors.org.  General  information 
about  USMA’s  Department  of  Mathemati¬ 
cal  Sciences  and  Department  of  Sys¬ 
tems  Engineering  can  be  found  at  http:// 
WWW . dean .u sma . edu/ math/ main .htm  and 
http://www.se.usma.edu,  respectively.  Some 
interesting  information  about  the  history  of 
the  mathematics  at  West  Point  is  posted  at 
http://www.dean.usma.edu/  math/history/ 
usma/index.htm.  If  you  have  any  questions 
regarding  the  site  of  the  67th  MORSS,  feel 
free  to  contact  the  site  coordinators  LTC 
Bob  Acker,  Department  of  Systems  Engi¬ 
neering,  email  fr5178@  usma.edu  or  voice 
(914)  938-5536,  or  LTC  Steve  Horton, 
Department  of  Mathematical  Sciences,  email 
as9492@usma.edu  or  voice  (914)  938- 
5905.0 


There  are  a  large  number  of  Special 
Sessions  scheduled  during  the  67th 
MORSS.  They  begin  at  1530 
daily,  and  will  be  held  in  either  auditori¬ 
ums  or  larger  rooms  in  order  to  accommo¬ 
date  everyone  who  will  want  to  attend 
them.  There  are  recurring  Special  Sessions 
with  which  many  of  you  may  be  familiar. 
These  include:  the  Rist  and  Barchi  Prize 
awards  and  papers  presentations, 
Junior/Senior  Analyst  Sessions,  the  Educa¬ 
tion  Session  and  reports  on  several  MORS 
Mini-Symposia  and  Workshops  held  over 
the  past  year.  In  addition,  there  are  three 
Special  Sessions,  one  each  day,  that  will  be 
unique  to  the  67th  MORSS  and  in  keeping 
with  this  year’s  theme  of  Focusing  Mili¬ 
tary  Operations  Research:  From  Our  Her¬ 
itage  to  the  Future. 

On  Tuesday,  22  June,  the  ongoing  quest 
for  coherent  “Theories  of  Combat”  that  can 
provide  descriptive  foundations  for  combat 
analysis  will  be  the  focus  of  a  Special  Ses¬ 
sion.  Papers  and  discussions  will  range 
from  early  attempts,  such  as  those  of  the 
Soviets  and  The  Military  Conflict  Institute, 
to  conduct  new  research  into  complexity, 
and  will  give  participants  an  appreciation 
for  why  it  is  so  important,  and  so  difficult. 


to  develop  a  self-consistent,  empirically- 
based  and  comprehensive  theory  against 
which  combat  models  can  be  compared. 
The  awarding  of  the  Rist  and  Barchi  Prizes 
and  presentation  of  the  winning  papers  and 
the  first  of  two  Junior/Senior  Analyst  ses¬ 
sions  are  the  two  other  Special  Sessions 
that  will  be  held  simultanedusly  on  Tues¬ 
day  afternoon. 

On  Wednesday,  23  June,  the  “Military 
Operations  Research  Heritage”  session  will 
focus  on  examining  some  classic  opera¬ 
tional  problems  from  each  of  the  services 
—  problems  that  have  been  addressed  but 
not  completely  solved  between  the  dawn 
of  the  modem  period  of  operational  analy¬ 
sis,  around  the  time  of  World  War  U,  and 
the  present.  These  problems,  one  example 
is  vulnerability  and  another  is  anti-subma¬ 
rine  warfare,  continue  to  challenge  military 
OR  analysts  even  today.  The  papers  and 
ensuing  discussion  will  afford  participants 
the  opportunity  to  convince  themselves 
that  even  modem  analytic  techniques  do 
not  preclude  the  obligation  we  analysts 
have  to  step  back  from  our  problems  on 
occasion  and  learn  from  our  able  predeces¬ 
sors.  Simultaneously  with  this  session  on 
(5^^  SESSIONS,/?.  20) 


DAY 

TIME 

TOPICS 

Tue 

1530-1715 

1.  Rist  &  Barchi  Prize  Awards/Prize  Papers 

2.  Junior/Senior  Analyst  #1 

3.  “Theories  of  Combat” 

Wed 

1530-1700 

1.  Junior/Senior  Analyst  #2 

2.  C'TSR  in  2010  Workshop/Joint  Experimentation 
Mini-Symposium  &  Workshop  Reports 

3.  “Military  OR  Heritage” 

Thu 

1530-1700 

1.  Education  Session 

2.  SIMTECH  2007  Mini-Symposium/SIMVAL  99 
Workshop  Reports 

3.  “The  Innovation  Process:  Warfighting  Advantage 
or  Achilles’  Heel?” 
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67th  MORS  SYMPOSIUM 


Junior/Senior  Analyst  Special  Session  for  the  67th  MORSS 


Mr.  Jay  Wilmeth 

SETA  Corporation 

e-mail:  wilmetjl@js.pentagon.mil 

The  Junior/Senior  Analyst  program 
will  take  place  for  the  tenth  con¬ 
secutive  year  at  the  67th  MORS 
Symposium  at  West  Point.  The  program 
conducted  during  Special  Session  periods 
will  take  on  a  somewhat  different  format 
this  year.  Historically,  this  event  of  the 
annual  Symposium  has  been  very  success¬ 
ful  and  has  drawn  both  junior  and  mid¬ 
level  audiences.  Since  the  theme  of  the 
67th  focuses  on  the  future  and  thus  the 
junior  analyst,  it  seems  appropriate  to  key 
the  activities  of  the  Junior/Senior  program 
toward  that  theme  while,  at  the  same  time, 
accommodating  the  mid-level  analysts 
who  desire  to  interact  with  the  seniors. 
With  that  in  mind,  the  board  of  directors 
chose  to  expand  the  scope  of  the  program 
this  year.  Accordingly,  there  will  be  two 
Junior/Senior  Sessions. 

The  first  session  should  attract  those 
MORSians  who  understandably  want  to 
meet  with  and  discuss  important  issues 
with  the  more  senior  analysts  known  to 
most  of  us.  This  session  is  scheduled  for 
Tuesday  afternoon  from  1530-1700,  in  an 
auditorium  that  will  accommodate  a  rela¬ 
tively  large  number  of  participants.  The 
session  will  be  open  to  all  to  hear  distin¬ 
guished  senior  analysts  discuss  topics 
relating  to  this  year’s  theme.  After  intro¬ 
ductory  remarks  from  each  of  the  seniors, 
the  balance  of  the  period  will  feature  a 
moderator-led  Q&A  session  from  the 
floor. 

Session  two,  scheduled  for  Wednesday 
afternoon  from  1530-1700,  in  smaller 
classrooms,  will  be  dedicated  exclusively 
to  the  junior  analyst.  We  define  junior 
analyst  as  one  who  is  relatively  new  to  the 
Military  Operations  Research  world  and 
will  benefit  from  the  wisdom  provided  by 
those  more  seasoned  analysts  who  have 
“been  around.”  It  will  follow  a  format 
that  has  been  successful  in  the  past.  There 
will  be  at  least  four  meeting  rooms  sepa¬ 
rated  along  Service  and  OSD  lines  featur¬ 
ing  mid-level  seniors  who  are  experienced 


analysts  and  who  are  familiar  with  the  day- 
to-day  problems  and  issues  facing  the  OR 
community.  The  sessions  will  feature  two 
seniors,  each  of  whom  will  address  the 
concerns  of  the  attendees.  It  is  envisioned 
that  the  questions  and  discussions  will  cen¬ 
ter  on  career  paths  within  the  Military  OR 


I  “A  complete  listof 
senior  analysts  participating  in 
both  ^ssions  will  be  published 
in  the  June  issue  of  the  PHALM.!' 

world  and  other  “Hot”  topics  important  to 
junior  analysts.  It  is  at  this  second  session 
that  junior  analysts  will  have  the  opportu¬ 
nity  to  meet  with  those  more  experienced 
analysts  who  are  currently  making  signifi- 


SESSIONS 

(continued  from  p.  19) 

Wednesday  afternoon,  the  second  Junior/ 
Senior  Analysts  session  will  be  held  and 
reports  on  the  C4ISR  in  2010  Workshop 
and  the  Joint  Experimentation  Mini-Sym¬ 
posium  and  Workshop  will  be  presented. 

On  Thursday,  24  June  a  panel  of  distin¬ 
guished  leaders  will  consider  “The  Innova¬ 
tion  Process:  Warfighting  Advantage  or 
Achilles’  Heel?”  This  process  develops 
new  military  technology,  doctrine  and  tac¬ 
tics,  and  transitions  them  to  the  warfighter. 
In  addition,  the  process  must  measure  its 
progress  against  the  anticipated  threat  and 
strive  for  an  optimal  allocation  of  scarce 


cant  contributions  to  military  analysis  and 
national  security  issues.  We  emphasize 
that,  while  all  are  welcome,  the  focus  will 
be  on  the  junior  analyst  during  the 
Wednesday  sessions.  The  sessions 
should  feature  no-holds-barred  discus¬ 
sions  and  lively  interchanges. 

A  complete  list  of  senior  analysts  par¬ 
ticipating  in  both  sessions  will  be  pub¬ 
lished  in  the  June  issue  of  the  PHALANX, 
Questions  and  suggestions  about  the  pro¬ 
gram  may  be  directed  to  the  program  co¬ 
chairs: 

Mr.  Jay  Wilmeth 
Land  &  Littoral  Warfare 
Assessment  Division 
J-8,  the  Joint  Staff 
Pentagon,  Washington,  DC  20318 
Voice:  703-695-4657 
Email :  wihnetjl  @ j s.pentagon.mil 

Mr.  Bill  Dunn 

US  Army  Model  and  Simulation  Office 
1 1 1 1  Jeff  Davis  Hwy .  Suite  503 
Arlington,  VA  22202 
Voice:  703-601-0011x25 
Email:  dunnwih@hqda.army.mil  O 


resources.  These  leaders  will  offer  the  lat¬ 
est  thinking  on  how  the  innovation  process 
seeks  to  maintain  American  military  domi¬ 
nance  during  this  time  of  rapidly  changing 
threats,  technologies  and  warfighting  con¬ 
cepts.  The  Education  Session  and  reports 
on  the  SIMTECH  2007  Mini-Symposium 
and  SIMVAL  99  Workshop  will  be  con¬ 
ducted  simultaneously  on  Thursday  after¬ 
noon. 

Your  participation  in  as  many  of  these 
Special  sessions  as  you  can  attend  is 
strongly  encouraged  —  see  the  reference 
guide  on  page  20. 0 
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September  1998  Recipient  of  the  MORS  Stephan  A.  Tisdale 
Graduate  Research  Prize  —  Major  Arent  Arntzen 


THESIS:  Software  Components  for  Air 
Defense  Planning 

AUTHOR:  Arent  Arntzen,  Major,  Norwe¬ 
gian  Air  Force  B.S.,  Norwegian  Air  Force 
Academy,  1991;  Master  of  Science  in 
Operations  Research-September  1998; 
Advisor  Arnold  Buss,  Department  of 
Operations  Research 

ABSTRACT: 

Modern  offensive  weapon  tech¬ 
nologies  such  as  stealth  and  pre¬ 
cision  guided  munitions  have 
rendered  Integrated  Air  Defense  Systems 
increasingly  vulnerable  and  ineffective. 
Stealth  effectively  reduces  the  performance 
of  radar,  but  does  not  have  the  same  impact 
on  passive  systems.  Sensors  have  been  the 
most  important  and  vulnerable  part  of  air 
defense  systems  throughout  the  history  of 
air  warfare.  Research  into  passive  sensors 
has  been  encouraging,  but  before  passive 
sensor  systems  are  produced,  procured  and 
deployed,  analysis  and  planning  must  be 
conducted  to  quantify  potential  benefit  and 
determine  feasible  system  configurations. 


As  this  type  of  analysis  encompasses 
extremely  complex  system  behavior,  devel¬ 
oping  reusable  and  flexible  simulation 
models  becomes  important.  This  thesis 
develops  a  prototype  software  component 
architecture  and  component  library  for 
building  simulation  models  for  air  defense 
analysis.  Sensor  and  airborne  weapon  simu¬ 
lation  components  are  models  for  air 
defense  analysis.  Sensor  and  airborne 
weapon  simulation  components  are  demon¬ 
strated  and  used  in  an  exploratory  analysis 
of  the  impact  of  a  network  of  Infrared 
Search  and  Track  sensors.  The  analysis  is 


based  on  a  modem  air  defense  system 
deployed  in  a  realistic  scenario.  The  com¬ 
ponent  architecture  and  documentation 
methodology  supports  reuse,  and  provides 
model  configuration  flexibility  with  poten¬ 
tial  for  growth  in  successive  stages  of 
analysis. 

Biography 

Major  Arntzen  was  bom  in  Olso,  Nor¬ 
way  in  1960,  He  joined  the  Royal  Norwe¬ 
gian  Air  Force  in  1979  and  completed  his 
Pilot  Training  with  the  United  States  Air 
Force  in  Texas  in  1 981,  Between  1981  and 
1991,  he  flew  the  F-5  and  F-16  aircraft  and 
spent  three  years  earning  his  Bachelors 
degree  at  the  Norwegian  Air  Force  Acade¬ 
my,  In  1991  he  went  back  to  the  Air  Force 
Academy  to  teach  Air  Power  history  and 
doctrine. 

Upon  graduation  from  the  Naval  Post¬ 
graduate  School,  Major  Arntzen  will  be 
promoted  to  Lt  Colonel  to  command  the 
operations  Group  of  Oerland  Main  Air 
Base.  The  unit  comprises  of  an  F-16 
squadron  and  supporting  units,  and  is  Nor¬ 
way's  Rapid  Reaction  Force  (Air),  O 


AFIT 1998  MORS  Prize  Recipient 


THESIS:  A  Value-Focused  Thinking 
Approach  to  Offensive  Information 
Operations 

AUTHOR:  Capt  Michael  P.  Doyle 
ABSTRACT: 

easures  of  merit  for  offensive 
information  operations  (lO)  are 
developed  and  applied  to  cam¬ 
paign-level  and  acquisitions  support  scenar¬ 
ios.  Value-Focused  Thinking  and  multiob¬ 
jective  value  analysis  are  employed  to 
develop  these  measures  of  merit  based  on 
the  values  and  preferences  of  owning  deci¬ 
sion  makers.  Using  these  measures  of  merit. 


courses  of  action  are  developed  as  part  of 
evaluating  campaign-level  planning.  These 
courses  of  action  are  scored,  evaluated,  and 
ranked  using  these  measures  of  merit.  Simi¬ 
larly,  the  10  weapon  systems  used  in  the 
acquisition  support  scenario  are  scored, 
evaluated,  and  ranked  based  on  these  mea¬ 
sures  of  merit.  The  sensitivity  of  the  resul¬ 
tant  alternative  ranking  is  evaluated  and  its 
importance  identified.  This  methodology  is 
also  used  to  characterize  each  alternative 
course  of  action  and  weapon  system  as  to  its 
ability  to  fulfill  the  owning  decision 
maker’s  objectives.  Finally,  these  character¬ 
istics  are  used  to  identify  opportunities  to 
develop  new  and  better  alternatives  for  each 

21 


decision  opportunity. 

Biography: 

Captain  Michael  P  Doyle  was  bom  on 
17  May  1964  in  Pennsylvania.  In  1982,  he 
graduated  from  both  the  Leechburg  Area 
High  School  and  the  Lenape  Area  Vocation¬ 
al  and  Technical  School  where  he  was 
nationally  ranked  in  chemical  technology. 
After  graduation,  he  attended  and  graduated 
from  the  Pennsylvania  State  University,  with 
Bachelor  of  Science  degree  in  Electrical 
Engineering.  While  attending  Penn  State  he 
also  managed  a  company  and  developed  a 

(See  AFIT,p.  25) 
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SIMVAL  99  —  Making  VV&A  Effective  and  Affordable 

Priscilla  Glasow,  Ml  I  KE  and  Dr.  Dale  Pace,  JHU/APL 
e-mail:  pglasow@iiiitre.org  and  dale.pace@jhuapl.edu 


Priscilla  Dr.  Dale  Pace, 

Glasow,  MITRE  JHU/APL 


The  Simulation  Validation  (SIM¬ 
VAL)  99  Workshop  was  held  at  The 
Johns  Hopkins  University  Applied 
Physics  Laboratory  on  26-29  January  1999. 
This  special  meeting  explored  ways  to 
ensure  Modeling  and  Simulation  (M&S) 
credibility  in  effective,  efficient,  and  afford¬ 
able  ways.  The  SIMVAL  99  Workshop 
was  co-sponsored  by  the  Military  Opera¬ 
tions  Research  Society  (MORS)  and  the 
Society  for  Computer  Simulation  (SCS) 
International,  and  was  the  fourth  in  the 
series  of  SIMVAL  workshops  sponsored  by 
MORS.  SIMVAL  99  examined  current 
technology  supporting  Verification,  Valida¬ 
tion  and  Accreditation  (W&A)  and  identi¬ 
fied  promising  areas  of  possible  technology 
development  that  could  enhance  VV&A. 
Such  enhancements  could  make  VV&A 
more  effective  for  a  given  level  of  resources 
or  could  allow  the  same  level  of  W&A  to 
be  achieved  with  reduced  resources.  The 
SIMVAL  99  Workshop  focused  on  three 
specific  areas:  (1)  verification  technology, 
(2)  validation  technology  and  methodology, 
and  (3)  the  impact  of  technology  on  W&A 
costs.  The  primary  objectives  of  SIMVAL 
99  were  to  provide  a  synopsis  of  the  state  of 
the  art  for  VV&A  technology  and  to  sug¬ 
gest  what  should  be  done  to  most  fully 
exploit  that  technology  and  advance 
W&A. 

The  mini-symposium  attracted  115  par¬ 
ticipants  from  government,  academia,  and 
industry.  Three  speakers  provided  a  focus 
for  the  workshop  discussions;  the  speakers 
and  their  topics  are  listed  at  the  end  of  this 
article.  Some  of  the  presentations  made  at 
SIMVAL  99  are  also  available  on  the 
MORS  web  site  at  (http://www.mors.org). 


Structure  and  Focus 

The  workshop  included  three  Working 
Groups  and  a  Synthesis  Group.  The  specific 
focus  of  each  Working  Group  is  further 
defined  in  the  Terms  of  Reference.  Each 
group  was  co-chaired  and  had  an  assigned 
reporter  to  capture  the  discussion  points. 
The  Working  Group  Co-Chairs  and 
Reporters  are  also  listed  at  the  end  of  this 
article. 

The  Workshop  was  used  to  test  two 
hypotheses.  The  first  hypothesis  was  that 
the  VV&A  community  is  generally  unin¬ 
formed  and  untrained  in  the  selection,  devel¬ 
opment  and  application  of  tools  and  tech¬ 
nologies  that  might  assist  in  the  conduct  of 
VV&A  activities.  The  second  hypothesis 
was  that  the  cost  of  performing  VV&A 
might  be  considerably  reduced  if  such  tools 
and  technologies  were  available  and  under¬ 
stood.  The  primary  goal  of  the  workshop 
then  was  to  provide  an  educational  forum  in 
which  participants  could  become  familiar 
with  existing  tools  and  technologies.  A 
unique  feature  of  this  special  meeting  was  a 
display  area  in  which  invited  vendors 
demonstrated  current  tools  and  technologies 
that  have  proven  useful  to  various  VV&A 
efforts.  The  workshop  was  also  intended  to 
provide  a  forum  in  which  additional  tools 
and  technologies  would  be  identified  by  the 
participants  and  examined  for  possible  appli¬ 
cation  to  other  W&A  endeavors. 

Workshop  ChaUenges 

The  workshop  participants  were  chal¬ 
lenged  in  three  ways.  First,  the  need  for 
education  about  tools  and  technologies  was 
highlighted  during  the  first  two  days  of  the 
workshop  as  participants  worked  to  focus  on 
tools  and  technologies.  This  challenge  was 
especially  strong  as  much  of  the  VV&A 
community  has  been  focused  on  definitions 
and  processes  during  recent  years,  rather 
than  tools  and  technologies.  Second,  the 
Working  Groups  stmggled  with  reaching  a 
common  understanding  from  among  the 
diverse  groups  represented  by  the  two  spon¬ 
soring  organizations  and  the  different  per¬ 
spectives  of  the  analytical,  acquisition,  and 
training  communities.  Third,  the  workshop 
format  was  unfamiliar  to  some  of  the  partici¬ 


pants  and  the  challenge  to  deliver  a  specified 
product  at  the  end  of  the  three  day  event 
required  re-thinking  of  previously-held 
methods  and  rapid  development  of  effective 
work  groups. 

Each  Working  Group  was  tasked  with 
developing  a  synopsis  of  the  key  points  of 
their  discussions  and  the  lessons  learned. 
Significant  issues,  concerns  and  recom¬ 
mended  solutions  regarding  the  use  of  tools 
and  technologies  to  support  VV&A  were 
also  identified  as  a  required  product  of  each 
group.  The  following  material  summarizes 
the  findings  and  conclusions  of  the  work¬ 
shop. 

Workshop  Findings 

The  workshop  findings  were  the  result  of 
the  Working  Groups’  discussions,  the  identi¬ 
fication  of  crosscutting  issues  by  the  Synthe¬ 
sis  Group  and  the  capstone  observations  of 
the  Workshop  Co-Chairs. 

•  The  Modeling  and  Simulation  (M&S) 
community,  and  the  subset  of  people  with¬ 
in  it  who  are  concerned  with  verification, 
validation,  and  accreditation,  is  so  broad 
and  diverse  that  effective  communication 
is  very  difficult.  Commonly  accepted  def¬ 
initions  for  VV&A  terms,  such  as  those 
provided  within  the  DoD  M&S  glossary 
and  in  formal  DoD  documents  such  as 
DoD  directives  and  the  DoD  WScA  Rec¬ 
ommended  Practices  Guide  (RPG),  can  be 
(and  are!)  interpreted  in  such  different 
ways  by  various  members  of  the  M&S 
community  that  misunderstandings  and 
communication  confusion  can  abound. 
Lack  of  a  clear  articulation  of  the  various 
perspectives  that  can  be  brought  to  M&S 
W&A  is  part  of  the  cause  for  this  prob¬ 
lem.  Those  working  with  software  archi¬ 
tectures  have  recognized  the  importance 
of  multiple  views  for  the  architecture  of  a 
software  development  to  provide  a  full 
description  of  the  planned  development, 
such  as  the  operational,  system  and  tech¬ 
nical  views  espoused  in  the  DoD  Joint 
Technical  Architecture.  M&S  W&A  has 
not  matured  to  the  place  of  having  formal¬ 
ized  the  different  perspectives  that  are 
equivalent  to  the  different  views  in  soft¬ 
ware  architecture.  Also  contributing  to 


PHALANX 


22 


March  1999 


communication  ambiguity  in  M&S 
W&A  is  the  lack  of  a  mathematical  level 
of  specificity  for  most  W&A-related  def¬ 
initions.  This  problem  manifested  itself 
significantly  in  Working  Group  1  (verifi¬ 
cation  technology)  and  impeded  their 
progress  until  the  group  decided  to  split 
and  tackle  verification  issues  independent¬ 
ly  from  a  global  perspective  and  from  a 
phase-specific  verification  process  per¬ 
spective.  It  is  believed  that  the  breadth 
and  depth  of  potential  trouble  that  such 
communication  difficulties  can  cause  the 
M&S  W&A  community  are  not  fully 
appreciated  by  either  W&A  practitioners 
or  M&S  management 

•  It  appears  that  the  VV&A  community  is 
not  exploiting  existing  technology  as 
much  as  desired.  The  reasons  for  this  are 
manifold.  First,  M&S  management  and 
YV&A  practitioners  as  a  whole  are  woe¬ 
fully  unaware  of  existing  tools  and  tech¬ 
nologies  that  could  be  used  to  support 
W&A.  Second,  the  W&A  community 
has  focused  primarily  to  date  on  defining 
terminology  and  developing  methodolo¬ 
gies  and  processes,  and  has  not  given  ade¬ 
quate  attention  to  the  potential  benefits  of 
tools  and  technologies.  Other  reasons 
include  the  lack  of  a  comprehensive  sur¬ 
vey  of  tools  and  technologies  available  to 
support  the  education  of  the  W^&A  com¬ 
munity  or  the  use  of  these  resources  in 
DoD  and  elsewhere.  No  central  reposito¬ 
ry  exists  to  document  W&A  tool  use  or 
to  serve  as  a  resource  for  future  applica¬ 
tions  of  W&A  tools  and  technologies. 
Consequently,  resources  to  support 
W&A  tool  use  are  not  identified  routine¬ 
ly  as  part  of  M&S  lifecycle  planning. 
Even  when  tools  are  used,  their  use  is 
often  ad  hoc  and  not  repeated  consistently 
from  one  M&S  project  to  the  next. 

•  It  appears  that  advancement  in  computa¬ 
tional  capabilities  and  software  engineer¬ 
ing  are  proceeding  more  rapidly  than  com¬ 
parable  advances  in  M&S  and  W&A, 
This  problem  is  exacerbated  by  the  limited 
awareness  of  M&S  management  and 
W&A  practitioners  about  the  potential 
current  and  evolving  technology  to  facili¬ 
tate  more  effective  and  affordable  W&A, 
However,  it  appears  that  the  time  may  be 
appropriate  for  major  advances  in  applica¬ 
tion  of  M&S  W&A  technology. 
Increased  emphasis  is  being  placed  upon 
M&S  use  in  system  design  and  acquisi¬ 
tion,  as  illustrated  by  the  Simulation  Based 


Acquisition  (SBA)  program,  and  upon 
model-based  decision  aids  for  operational 
planning  and  execution.  The  Department 
of  Energy  (DOE)  Accelerated  Strategic 
Computing  Initiative  (ASCI)  program 
includes  a  major  software  development 
component  (with  some  emphasis  on  veri¬ 
fication  and  validation  of  codes  used  in 
computational  science  and  other  areas  of 
“grand  challenge”  problems)  as  well  as 
the  ASCI  emphasis  on  advances  in  com¬ 
putational  capabilities.  Continuing  com¬ 
putational  advances  and  improvements  in 
software  development  environments 
make  automation  of  M&S  V&V  more 
viable  just  as  they  have  already  demon¬ 
strated  such  automation  viability  in  soft¬ 
ware  V&V.  However,  it  should  be  noted 
that  procurement  officials,  project  man¬ 
agers,  and  planners  also  have  to  be  aware 
of  the  benefits  of  leveraging  tools  used  in 
M&S  development  for  V&V  efforts  so 
that  contracts,  etc.  will  be  written  to  effec¬ 
tively  accommodate  exchange  and  shar¬ 
ing  of  information,  data,  and  tools  neces- 
saiy  for  these  benefits  to  be  realized. 

•  VV&A  tools  and  technologies  should  be 
able  to  reduce  program  cost  and  risk  if 
used  properly.  However,  the  importance 
of  making  the  necessary  investment  to 
enable  use  of  such  M&S  W&A  tools  and 
technologies  has  not  been  widely  recog¬ 
nized  within  DoD  or  elsewhere.  In  part, 
this  situation  exists  because  there  is  very 
little  reliable  cost  benefit  information 
available  relative  to  M&S  W&A.  Avail¬ 
able  cost  information  is  not  well  orga¬ 
nized  for  effective  analysis  (no  standard 
cost  element  identification  exists,  useful 
and  widely  accepted  metrics  do  not  exist, 
etc.).  This  reduces  the  basis  for  cost  bene¬ 
fit  assessments  to  either  mere  postulation 
or  to  argument  by  analogy  through  com¬ 
parisons  between  the  dearth  of  tools  used 
to  support  M&S  VV&A  with  the  more 
extensive  application  of  automated  tools 
in  the  software  V&V  arena. 

•  It  appears  that  commercial  forces  will  con¬ 
tinue  to  drive  the  development  of  software 
V&V  technology  more  rapidly  than  M&S 
management  and  VV&A  personnel  are 
likely  to  assimilate  and  exploit  fully. 
However,  that  may  not  be  the  case  for 
technology  that  is  peculiar  to  M&S 
VV&A  that  is  otherwise  not  generally 
applicable  for  software  V&V.  The  stimu¬ 
lus  of  government  Research  and  Develop¬ 
ment  (R&D)  funding  may  be  required  for 
needed  progress  in  this  area. 


•  A  theme  that  was  prominent  in  many  parts 
of  the  workshop  is  the  importance  of  tal¬ 
ented  people.  Tools  and  technology  can 
supplement,  but  will  not  replace  the 
important  role  that  knowledgeable  people 
have  in  effective  VV&A. 

Recommendations 

•  There  needs  to  be  a  sustained  educational 
campaign  to  ensure  that  M&S  managers 
appreciate  the  importance  of  VV&A, 
understand  the  cost-benefit  potential  of 
VV&A  for  their  applications  and  under¬ 
stand  that  the  resources  required  to  exploit 
W&A  technology  in  ways  that  are  both 
effective  and  affordable.  This  educational 
campaign  must  also  ensure  that  VV&A 
practitioners  are  aware  of  available 
methodologies,  techniques,  tools,  tech¬ 
nologies  and  the  implications  of  their 
application  or  non-application.  No  single 
organization  is  likely  to  stimulate  or  pro¬ 
vide  the  total  impetus  needed  for  such  an 
educational  campaign;  instead  each  of  the 
MORS  Sponsors,  SCS  leadership,  DMSO 
and  others  are  encouraged  to  establish 
educational  campaigns  within  their 
spheres  of  influence  or  provide  the  neces¬ 
sary  resources  to  allow  their  people  to 
benefit  from  the  educational  campaigns  of 
other  organizations.  Such  educational 
campaigns  must  be  multi-faceted,  not  only 
addressing  audiences  of  VV&A  practi¬ 
tioners  through  short-courses,  conference 
presentations  and  articles  in  peer-reviewed 
trade  journals,  but  also  addressing  M&S 
managers. 

•  Several  areas  of  research  are  required.  At 
the  most  fundamental  level,  a  relatively 
comprehensive  survey  of  available 
W&A  tools  and  technologies  is  required. 
This  survey  should  address  how  such  tools 
and  technologies  should  be  used  properly, 
identify  appropriate  applications  and  link 
tool  use  to  phases  of  the  M&S  lifecycle. 
Information  about  resource  requirements 
for  tool  use  is  also  needed,  such  as  data, 
personnel,  training,  equipment  and  finan¬ 
cial  considerations.  Research  should 
include  establishment  of  a  repository  of 
VV&A  tool  information  that  identifies 
prior  tool  uses,  results  of  tool  application 
and  costs  of  use.  This  repository  may  be 
either  new  or  included  within  an  existing 
repository.  In  addition,  some  level  of  sup¬ 
port  for  basic  research  related  to  W&A  is 
indicated.  This  arena  has  been  neglected 

(See SJMyAL99, p.  24) 
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SIMVAL99 

(continued from  p.  23) 
to  date,  and  such  investment  is  needed  if 
the  long-term  promise  of  VV&A 
improvements  from  technology  advance¬ 
ments  are  to  be  fully  realized.  The  devel¬ 
opment  of  a  formalized  methodology  is 
also  needed  to  assist  users  in  determining 
how  good  a  simulation  must  be  if  it  is  to 
support  a  particular  kind  of  application. 
When  such  guidance  can  be  reduced  to 
formalized  methodology,  it  becomes 
amenable  to  automation,  with  the  conse¬ 
quence  of  enhanced  W&A  effectiveness 
concurrent  with  reduction  in  the  cost  of 
performing  W&A.  In  addition  to  DoD 
investments  in  research,  agencies  external 
to  DoD  that  support  basic  M&S  research, 
such  as  the  National  Science  Foundation 
(NSF),  other  government  agencies,  com¬ 
mercial  enterprises,  or  private  foundations, 
should  consider  the  urgent  need  for  more 
rapid  advancement  of  W&A  methodolo¬ 
gies  and  technologies  when  dispersing 
their  funds.  A  corollary  to  the  above  is  the 
necessity  for  sharing  information  devel¬ 
oped  by  such  research  throughout  the 
M&S  W&A  community 

•  Development  of  a  widely  accepted  and 
widely  used  way  to  identify  and  describe 
different  W^&A  perspectives  is  also  need¬ 
ed  to  reduce  communication  confusion 
among  disparate  elements  of  the  M&S 
and  W&A  communities.  Concomitant 
with  development  of  such  standard  per¬ 
spectives  (views)  would  be  elaboration  of 
VV&A-related  terminology  to  provide 
greater  precision  of  meaning.  Specifica¬ 
tion  of  W&A  cost  elements  must  be 
included  in  this  work,  so  that  meaningful 
W&A  cost  data  can  be  captured  to  pro¬ 
vide  a  factual  basis  for  development  or 
refinement  of  W^&A  cost  models. 

In  addition  to  the  above  general  recom¬ 
mendations,  the  M&S  W&A  community 
should  also  give  attention  to  the  more  specif¬ 
ic  recommendations  contained  within  the 
synopses  from  the  three  working  groups  and 
the  Synthesis  Group. 

Tool  Selection  and  EfBciencies  of  Use 

The  workshop  participants  found  that 
tools  and  technologies  selected  for  use  in 
conjunction  with  W&A  will  depend  on 
many  factors,  including  the  intended  tool 
use,  the  criticality  of  the  application,  the 
maturity  of  the  product,  corporate  culture 
and  the  type  of  M&S  development  para¬ 


digm.  The  cost  of  using  tools  and  technolo¬ 
gies  similarly  depend  on  these  factors,  as 
well  as  the  stability  and  complexity  of  user 
requirements,  security  requirements,  the 
time  available  in  which  to  use  technology, 
the  probability  of  risk  and  its  impact  and  the 
availability  of  reference  data.  The  compe¬ 
tence  of  W&A  agents  was  also  noted  as  a 
factor  that  can  influence  the  degree  and 
quahty  to  which  tools  and  technologies  are 
used. 

Existing  Tools  and  Technology 

Working  Group  1  examined  the  use  of 
tools  and  technologies  to  support  verifica¬ 
tion.  This  group  stated  that  comprehensive 
computer  automated  support  tools  would  be 
useful  to  support  requirements  verification. 
They  also  found  that  conceptual  models 
require  tools  that  promote  a  standard 
approach  for  development.  Working  Group 
1  determined  that  existing  tools  are  suffi¬ 
cient  for  the  design  and  coding  phases  of 
M&S  development  and  verification. 

Working  Group  2  focused  on  validation 
technology  and  methodology.  They  claimed 
that  it  is  easy  to  derive  desired  properties  for 
validation  tools,  but  difficult  to  find  tools 
that  demonstrate  these  properties.  Working 
Group  2  noted  that  improvements  are  need¬ 
ed  for  tools  that  support  the  development  of 
model  validation  criteria  and  validation  of 
the  conceptual  model.  They  identified  exist¬ 
ing  tools  that  can  be  used  to  support  data 
validation,  including  database  management 
systems,  data  quality  engineering  technolo¬ 
gy  and  data  modeling  tools.  Although  these 
tools  are  generally  mature  for  software  engi¬ 
neering  purposes,  they  do  not  adequately 
support  M&S  validation  activities.  Similar¬ 
ly,  the  group  identified  statistical  analysis, 
visualization,  and  after  action  tools  that  are 
useful  for  validation,  but  which  require  an 
improved  focus  to  meet  validation  needs. 

Working  Group  3  identified  existing 
tools  and  technologies  and  assessed  their 
impact  on  W&A  costs.  This  group  also 
identified  some  of  the  primary  factors  that 
affect  W&A  costs  and  examined  various 
technologies,  including  high  order  lan¬ 
guages,  integrated  development  environ¬ 
ments  and  synthetic  environments.  They 
found  that  program  management  tools,  auto¬ 
mated  repositories  and  database  technolo¬ 
gies  offer  the  greatest  potential  for  cost  sav¬ 
ing  through  the  sharing  and  increased  access 
to  program  information.  Conversely,  securi¬ 
ty  technology  was  one  example  of  technolo¬ 
gy  that  could  reduce  or  increase  W&A 


costs,  depending  on  the  degree  to  which  the 
technology  improved  information  sharing  or 
limited  access  to  data.  The  use  of  technolo¬ 
gies  to  support  W&A  was  impacted  most 
significantly  by  the  amount  of  time  available 
for  its  use.  Working  Group  3  also  found  that 
the  availability  of  reference  data  was  impor¬ 
tant  to  the  use  of  tools  and  technology. 

Technology  Recommendations 

The  Working  Groups  recommended 
leveraging  instrumentation  technology  to 
generate  real  or  experimental  data  against 
which  simulations  can  be  validated.  They 
observed  that  tools  are  also  needed  to  vali¬ 
date  human  behavior  in  interactive  simula¬ 
tions  and  where  human  behavior  is  repre¬ 
sented  in  M&S.  Knowledge  acquisition 
and  engineering  tools  were  identified  as  a 
far-term  need,  to  support  the  translation  of 
real  world  semantics  to  a  conceptual  model 
syntax.  The  Synthesis  Group  further  stated 
the  need  for  expressive  high-level  languages 
and  a  common  conceptual  model  to  facili¬ 
tate  effective  communication  between  soft¬ 
ware  engineers  and  subject  matter  experts. 

Methodology  Recommendations 

Although  the  workshop  focused  primari¬ 
ly  on  tools  and  technology.  Working  Group 
2  was  also  tasked  with  examining  the  inter¬ 
play  of  validation  methodologies  with  tech¬ 
nology.  The  group  offered  a  variety  of  rec¬ 
ommendations  for  the  near-term,  including 
the  need  for  independent  validation  of  con¬ 
ceptual  models  and  the  extension  of  valida¬ 
tion  methodologies  to  include  simulation 
based  acquisition  and  joint  experimentation 
requirements.  Their  far-term  recommenda¬ 
tions  included  the  need  to  better  understand 
modeling  theory  and  complexity  to  ensure 
that  technologies  employed  are  theoretically 
based  and  meet  the  needs  of  complex  sys¬ 
tems  and  environments. 

The  Synthesis  Group  also  recommended 
that  integrating  methodologies  are  needed  to 
allow  the  synthesis  of  W&A  activities  into 
coherent  metrics.  Users  can  then  apply  such 
metrics  to  understand  the  decision  risks 
associated  with  a  given  model  and  its  intend¬ 
ed  use.  The  Synthesis  Group  also  recog¬ 
nized  the  need  for  a  common  model  for 
assessing  the  value  of  W&A  activities, 
especially  hidden  and  deferred  costs. 

Conclusions 

Tools  and  technologies  currently  exist 
that  could  significantly  enhance  the  conduct 


PHALANX 


24 


March  1999 


of  VV&A  activities,  but  are  not  widely 
employed  due  to  a  lack  of  knowledge  and 
understanding  of  their  capabilities  and 
applicability  to  W&A.  The  MORS  com¬ 
munity  must  address  these  needs  to  attain 
the  greater  efficiencies  and  improved  effec¬ 
tiveness  that  are  offered  by  such  tools  and 
technologies.  We  must  move  beyond  our 
current  fascination  with  definitions  and 
process  diagrams,  and  focus  on  responsibly 
seeking  new  ways  to  reduce  costs  and  risks 
to  DoD  programs.  The  legacy  of  MORS  in 
the  W&A  arena  is  strong  and  our  commu¬ 
nity  can  again  lead  the  practice  forward. 

SIMVAL  99  Workshop  Co-Chairs 

Mrs.  Priscilla  A.  Glasow,  The  MITRE  Cor¬ 
poration 

Dr.  Dale  K.  Pace,  Johns  Hopkins  University 
Applied  Physics  Laboratory 

Mini-Symposium  Speakers 

CAPT  Dennis  K.  McBride,  USN,  Office  of 
Naval  Research  —  “The  Future  of  Tech¬ 
nology  and  W&A  in  Warfighting  Simu¬ 
lation” 

COL  Kenneth  Konwin,  USAF,  Director, 
Defense  Modeling  and  Simulation  Office 
—  “V&V  Technology  Review” 

Dr.  Paul  K.  Davis,  RAND  —  “Beyond  Box 
Checking:  How  Can  Technology  Help 
Model  Building  and  W&A?” 

Working  Group 
Co-Chairs  and  Reporters 

Working  Group  1,  Verification 
Technology  Co-Chairs: 

Dr.  Osman  Balci,  Mr.  Robert  Lewis 
Reporter:  Ms.  Susan  Solick 

Working  Group  2,  Validation  Technology 
and  Methodology  Co-Chairs: 

Mr.  Thomas  Ruth,  Mr.  Gary  Coe 
Reporter:  Mr.  Dennis  Laack 

Working  Group  3, 

W&A  Costs  Co-Chairs: 

Mr.  Richard  Kuhn,  Mr.  William  Waite 
Reporter:  Mr.  William  Jordan 

Synthesis  Group 

Co-Chairs:  Dr.  Jack  Morrison,  Ms. 

Simone  Youngblood 
Reporter:  Dr.  Walt  Stanley 
For  an  extensive  bibliography,  please  refer 
to  the  recommended  readings  listed  in  the 
Terms  of  Reference  and  to  the  read-ahead 
package  located  on  the  MORS  web  site  at 
(http://www.mors.org).  O 
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designed,  coded,  tested  and  fielded.  While 
the  testing  and  fielding  is  being  done,  the 
specification  of  the  next  phase  is  underway. 
By  managing  the  parallelism  and  feedback, 
several  concurrent  developmental  iterations 
can  help  define  and  include  the  true 
requirements.  By  building  smaller  pieces 
and  using  the  iterations  to  help,  JSIMS  is 
being  built  to  satisfy  the  user’s  needs,  even 
if  they  were  unable  to  articulate  them  up 
front. 

Summary 

JSEMS  will  support  the  entire  training, 
rehearsal,  doctrine  development,  and  pro¬ 
fessional  mihtary  education  process  by  pro¬ 
viding  a  set  of  integrated  and  flexible  tools 
that  can  be  composed  based  upon  the  needs 
of  the  exercise.  In  doing  so,  the  JSIMS 
events  can  be  distributed  locally  across 
machines  and  distantly  across  theatres 
while  interoperating  with  live,  virtual  and 
constructive  participants.  JSIMS  will 
replace  numerous,  redundant  modeling  and 
simulation  systems  that  are  not  interopera¬ 
ble.  JSIMS  is  being  designed  and  built  to 
mate  with  Go-to-War  C4I  systems  and  pre¬ 
pare  commanders  to  coordinate  their  forces 
for  whatever  they  might  face  in  the  future. 

More  information  on  the  programmatics 
and  the  challenges  facing  JSIMS  can  be 
found  on  the  JSIMS  homepage  at 
www.jsims.mil. 

Notes 

a.  Interestingly  enough,  the  six  independent 
contracting  actions  led  to  six  different 


prime  contractors.  The  Navy  elected  to 
develop  the  software  in  house. 
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database  system  for  the  Allegheny  Ludlum 
Steel  Corporation  that  is  still  used  through¬ 
out  the  northeast.  After  graduation,  he  mar¬ 
ried  his  loving  wife  Barbara  before  entering 
active  duty  in  the  Air  Force  via  Officer 
Training  School.  His  first  assignment  was 
as  a  Minuteman  ICBM  Launch  Control  offi¬ 
cer  where  he  quickly  rose  to  Flight  Lead. 
During  this  tour  he  completed  a  Master  of 
Science  Degree  in  Space  Studies  at  the  Uni¬ 
versity  of  North  Dakota.  His  second  assign¬ 
ment  took  him  to  the  National  Air  Intelli¬ 


gence  Center  (NAIC)  where  he  served  as 
the  Air  Force  and  the  Defense  Intelligence 
Agency's  national  technical  expert  on  aero¬ 
space  electrical  power  conversion,  genera¬ 
tion,  and  storage.  He  completed  his  tour  at 
NAIC  as  Deputy  Chief  of  the  Advanced 
Power  and  Weapons  Technology  Branch. 
Capt  Doyle's  third  assignment  was  to  the 
Air  Force  Institute  of  Technology,  where  he 
graduated  in  March,  1998  with  a  master's 
degree  in  Operations  Research.  Captain 
Doyle  is  currently  assigned  to  the  United 
States  Strategic  Command  (USSTRAT- 
COM),  Offut  AFB,  NB;  as  an  operations 
research  analyst.  O 
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(continued  from  p.  1) 

These  criteria  were  applied  to  over  4000 
courses,  which  resulted  in  initial  identifica¬ 
tion  of  1250  courses  with  technology  reengi¬ 
neering  potential. 

These  1250  courses  were  then  subjected 
to  a  more  rigorous  analysis.  Consideration 
was  given  to  several  additional  factors, 
including: 

•  Whether  the  course  was  a  part  of  a 
pipeline; 

•  Whether  there  were  substantial  changes 
to  the  current  course  curriculum  planned 
within  the  next  year; 

•  If  the  course  was  part  of  a  planned  rating 
merger;  and, 

•  What  was  the  criticality  of  the  rating 
being  trained  with  respect  to  fleet 
requirements  and  manning  level. 

Commercial  off-the-shelf  and  govern¬ 
ment  developed  media  analysis  software 
was  applied  to  the  courses.  This  software 
allowed  us  to  determine  the  most  optimum 
delivery  methods  for  the  course  of  instruc¬ 
tion  that  factored  in  cost  of  delivery  method 
and  projected  student  success  rates. 

The  next  step  was  to  determine,  course 
by  course,  where  the  training  benefits  would 
occur  and  how  to  account  for  the  benefits  on 
a  cost  basis.  For  example,  both  private  sec¬ 
tor  and  our  own  analysis  indicated  that 
applying  enhanced  electronic  instruction  to 
selected  curriculum  would  reduce  time  to 
train.  This  reduction  is  gained  principally  by 
increasing  the  amount  of  visual  learning 
through  improved  graphics  and  animations 
that  the  new  technology  makes  possible.  A 
course  that  is  currently  138  days  in  length 
with  a  annual  throughput  of  2500  students, 
obtaining  a  15%  course  reduction  (the  low 
end  of  expected  results)  can  reduce  training 
man-year  costs  by  400  thousand  dollars 
annually.  A  typical  course  delivered  via 
Video  TeleTraining  (VTT)  can  reduce  travel 
costs  by  25-30  thousand  dollars  annually. 

All  the  courses  were  then  prioritized 
in  terms  of  return  on  investment.  These 
included: 

•  Calculating  the  expected  reductions  in 
total  time  to  train; 

•  Attrition  and  setback  rates; 

•  Temporary  duty  cost  savings;  and, 

•  Increases  in  throughput  as  a  result  of 
reduced  total  time  to  train. 

The  increased  throughput  without  a  cor¬ 
responding  increase  in  infrastructure  is  espe¬ 


cially  critical  for  high  demand  ratings.  Dur¬ 
ing  Fiscal  Year  1998  for  example,  CNET 
was  able  to  train  an  additional  300  Fire  Con- 
trolman  because  of  the  electronic  trainer 
used  in  the  Advanced  Electronic  Technical 
Core.  Students  using  this  method  of  training 
completed  training,  on  average,  in  less  than 
100  days  compared  to  the  137  days  for  tradi¬ 
tional  students. 

The  next  step  was  to  go  beyond  technolo¬ 
gy  reengineering  and  identify  advanced 
courses  that  had  high  compatibility  with 
civilian  training.  This  part  of  the  study  also 
focused  on  courses  in  highly  technical  areas 
where  the  Navy  was  facing  challenges  in 
delivering  state-of-the-art  training  required 
to  support  state-of-the-art  systems.  One 
caveat,  however,  was  not  to  include  any 
direct  combat  systems  training.  In  the  first 
category  of  civilian  equivalence,  over  50 
courses  in  the  fields  of  welding,  food  prepa¬ 
ration,  small  engine  repair,  barbering, 
machinery  repair,  supply  and  hazardous 
material  were  identified  as  having  the  poten¬ 
tial  to  be  trained  in  the  private  sector.  After 
identification  of  the  courses,  research  was 
performed  to  determine  private  sector 
providers  who  could  likely  provide  the  need¬ 
ed  training  at  less  cost  than  doing  it  in- 
house.  These  vendors  were  then  catego¬ 
rized:  community  colleges,  technical 
schools,  industrial  sites  (both  government 
and  private)  and  traditional  contract  training. 
It  was  determined,  for  example,  that  locating 
an  advanced  culinary  course  in  Norfolk 
(factoring  in  tuition  and  other  costs)  will 
result  in  a  savings  of  900  thousand  dollars 
per  year  in  manpower  and  travel  costs. 

With  respect  to  improving  capability  to 
deliver  state-of-the-art  training,  the  field  ini¬ 
tially  chosen  for  review,  based  on  require¬ 
ments  and  analysis  of  existing  training,  was 
information  systems.  This  analysis  has 
resulted  in  CNET  contracting  a  demonstra¬ 
tion  project  that  makes  available  via  the 
Internet  over  400  information  systems  cours¬ 
es,  covering  topics  from  basic  word  process¬ 
ing  to  beginning  router  configurations.  Ini¬ 
tial  response  has  been  extremely  favorable. 
If  projections  continue  at  the  current  pace, 
when  fully  implemented  the  Navy  can 
expect  to  save  over  five  million  dollars  a 
year  in  training  and  travel  costs.  The  Navy 
is  also  piloting  an  advanced  networking 
course  using  a  private  technical  school.  The 
management  of  networks  both  afloat  and 
ashore  is  a  critical  requirement  in  ensuring 
information  warfare  dominance.  In  addition 
to  reducing  the  cost  of  training,  we  believe 


this  will  also  provide  better  and  more  timely 
training.  This  approach  provides  the  trainer 
another  tool  in  responding  to  the  increased 
commercial  off-the-shelf  procurements  and 
allows  the  delivery  of  training  to  fleet  con¬ 
centration  areas. 

As  a  direct  result  of  these  studies  and 
analysis  efforts,  CNET  has  established  a 
Local  Training  Authority  (LTA)  in  every 
CONUS  Fleet  concentration  area.  The  LTA 
is  responsible  for  brokering  training  in  their 
area.  More  directly,  they  locate  the  most 
effective,  efficient,  and  economical  training 
source/vendor  for  the  customer.  In  addition, 
he  must  also  work  with  the  fleet  customer  to 
identify  additional  courses  for  reengineering. 

The  culmination  of  the  first  phase  of 
analysis  was  the  Program  Objectives  Memo¬ 
randum  2000  (POM  00)  training  assessment. 
This  assessment  provided  the  detailed  fiscal 
analysis  to  support  the  reengineering  plan.  It 
presented  to  Navy  leadership  in  the  form  of 
the  training  baseline  memorandum  a  five- 
year  strategy  to  modernize  the  initial  training 
system.  This  includes  an  affordable  means 
of  paying  for  the  required  investment  and 
results  in  over  200  million  dollars  in  savings 
to  the  Navy  by  year  2005.  This  plan  was 
70%  funded  by  the  Navy. 

This  assessment,  presented  to  the  Navy’s 
operators,  is  designed  to  deliver  Sailors  to 
the  fleet  faster  while  maintaining  or  improv¬ 
ing  the  quality  of  Navy  training.  It  is 
designed  to  increase  advanced  training  in  the 
fleet  concentration  areas  meaning  Sailors 
will  spend  less  time  away  from  their  home- 
ports  and  families.  It  is  also  an  opportunity 
for  pull  down  enhancement/  sustainment 
training.  Today,  the  Navy  has  over  70 
courses  available  via  video  teletraining. 
There  are  currently  79  Learning  Resource 
Centers  (LRCs)  supporting  over  184  cours¬ 
es,  50  Automated  Electronic  Classrooms 
(AECs)  supporting  over  50  courses,  over 
400  courses  available  via  the  Internet  and, 
most  importantly,  a  funded  plan  that  will 
reengineer  individual  training  and  keep  the 
Navy  producing  the  quality  Sailors  it  needs 
into  the  21st  Century.  Solid  research  based 
on  analytical  planning  and  a  willingness  to 
look  outside  the  box,  were  the  engines  that 
drove  the  change.  Those  same  engines  will 
continue  to  guide  the  course. 
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STABILITY 

(continued  from  p.  8) 

Another  question  that  arises  is:  If  solu¬ 
tions  obtained  by  means  of  genetic  algo¬ 
rithms  are  indeed  more  stable  than  those 
derived  by  conventional,  optimal  seeking 
methods,  why  is  this  so?  It  is  my  guess  that 
the  inherent  stability  of  the  solutions 
derived  by  algorithms  that  emulate  evolu¬ 
tion  is  due  to  the  implicit  role  that  stability 
plays  in  evolution.  As  populations  (e.g.,  of 
plants  and  animals)  evolve,  they  invariably 
tend  toward  stability  as  well  as  fitness. 
Mother  Nature  simply  does  not  tolerate 
highly  unstable  populations  of  any  species. 
By  mimicking  the  evolutionary  process,  it 
would  seem  that  genetic  algorithms  also 
provide  solutions  that  are  inherently  stable. 
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work.  This  material  needs  to  span  the  theo¬ 
retical,  practical,  managerial  and  organiza¬ 
tional.  We  have  already  begun  this.  Our 
IMAS  meeting  produced  proceedings  that 
we  are  currently  distributing  to  the  attendees 
and  will  shortly  be  economically  available 
for  purchase  by  everyone.  Dr.  J.  P.  Bal- 
lenger,  our  Vice  President,  has  begun  to 
collect  a  volume  of  essays  from  our  seniors 
on  the  future  needs  of  our  profession,  tenta¬ 
tively  entitled  War  After  Next.  Dr.  Jim 
Taylor,  NPS,  has  developed  some  intrigu¬ 
ing  panel  discussions  for  the  Cincinnati 
meeting,  and  we  hope  to  have  a  MAS  Ple¬ 
nary  there.  Also,  we  are  planning  some 
educational  sessions  at  2MAS. 

In  future  columns:  First  MAS  Survey 
and  details  of  2MAS.  O 
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REVOLUTIONS 

(continued  from  p.  10) 

Means  Objectives  and  Decisions 

Means  objectives  describe  how  funda¬ 
mental  objectives  are  achieved.  In  the  case 
of  Joint  Vision  2010  the  overall  means 
objective,  “Full  Spectrum  Dominance”  is 
described  in  the  joint  literature.^  It  has  four 
operational  concepts:  dominant  maneuver, 
precision  engagement,  full  dimensional 
protection  and  focused  logistics.  Measur¬ 
ing  outcomes  with  respect  to  each  of  these 
operational  concepts  could  be  thought  of  as 
evaluating  means  objectives.  Figure  3  pre¬ 
sents  these  objectives  and  some  potential 
measures  that  are  indicative  of  success 
along  each  of  these  concepts.  There  are 
three  things  that  become  apparent  as  one 
reviews  the  means  measures  that  are  pre¬ 
sented.  First,  in  this  problem  domain,  the 
intersection  with  measures  describing  fun¬ 
damental  objectives  is  small.  Second,  suc¬ 
cess  among  these  measures  is  highly  inter¬ 
dependent.  Dominant  maneuver,  as  an 
example,  requires  accurate  information 
about  an  enemy’s  location  and  sufficient 
logistical  resources  to  support  unit  activi¬ 
ties.  Third,  information  related  measures 
of  effectiveness  (C^ISR  measures),  the 
heart  of  JV  2010,  are  not  evaluated  direct¬ 
ly.  This  is  because  information  derives  its 
value  in  the  context  of  some  set  of  deci¬ 


sions  or  actions,  and  information  is  a  com¬ 
modity  used  by  commanders  to  achieve 
fundamental  objectives. 

Computer-based  simulations  of  combat 
and  other  battlefield  activities  are  among 
the  most  common  tools  that  are  used  to 
assess  effectiveness  along  these  means 
measures.  We  are  keenly  aware  of  the 
many  specialized  simulations  that  help 
decision  makers  “see  the  value”  of  choos¬ 
ing  a  particular  investment  strategy  or 


course  of  action.  The  strengths  and  weak¬ 
nesses  of  the  current  generation  of  tools  are 
discussed  heavily  in  the  literature.^  In 
addition  to  the  known  weaknesses  in  the 
tools,  there  is  strong  evidence  that  the  theo¬ 
retical  foundation  supporting  current  com¬ 
bat  models  no  longer  adequately  abstracts 
the  key  processes  that  occur  on  the  battle¬ 
field,  This  is  particularly  true  of  the 
decision  making  component  of  Command 
and  Control  (C2)  and  the  required  mathe¬ 
matical  models  supporting  automated  rea¬ 
soning  under  uncertainty. 

Addressing  Uncertainty 

There  are  many  sources  of  uncertainty 
that  affect  success  in  warfare.  There  is 
general  agreement  that  these  uncertainties 
affect  the  quality  of  analytic  insights  that 
are  provided  to  decision  makers.  For 
example,  there  is  clear  evidence  that  small 
changes  in  combat  simulation  inputs  can 
cause  very  significant,  unexpected  differ¬ 
ences  in  outcomes.  The  implications  of 
uncertainty  on  the  policy  development 
process  are  also  being  discussed  in  the  lit¬ 
erature.^^  The  analytic  community  sees 
the  implications  of  uncertainty,  and  the 
computational  technology  is  quickly 
emerging  to  efficiently  address  many  of  the 

uncertainties.  Id 

A  common  misperception  is  that  critical 
uncertainties  can  be  addressed  by  develop¬ 
ing  higher  resolution  models  of  combat 
processes.  Higher  fidelity  models  provide 
the  ability  to  increase  the  “realism”  of  a 
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Figure  3.  Measures  for  means  objectives 


PHALANX 


28 


March  1999 


model.  This  realism  can  be  beneficial  for 
visually  displaying  results  to  decision  mak¬ 
ers.  It  is  also  useful  for  differentiating 
between  alternative  weapon  systems  within 
a  small  outcome  space  and  resolving  low- 
level  system  engineering  type  uncertain¬ 
ties.  Additionally,  increased  fidelity  also 
allows  for  realistic  displays  that  are  very 
valuable  in  training  applications.  Yet,  this 
fidelity  also  requires  “certain”  data  that  are 
often  not  possible  to  produce  because  the 
required  information  is  unknowable. 
Rather  than  looking  to  increase  the  fidelity 
of  uncertain  data,  analysts  and  the  policy 
makers  they  support  would  be  better 
served  by  increasing  the  breadth  of  their 
perspective.  This  is  accomplishable 
through  the  development  of  a  systematic, 
consistent  frame  of  reference  for  communi¬ 
cating  clearly  with  decision  makers  con¬ 
cerning  the  sources  and  implications  of  the 
key  uncertainties  that  affect  outcomes 
along  fundamental  objectives. 

A  possible  frame  of  reference  for 
decomposing  the  sources  of  uncertainty 
consists  of  four  components: 

1)  Uncertainty  in  the  scenario; 

2)  Uncertainty  in  an  opponent’s  behavior; 

3)  Uncertainty  in  technology  performance; 

and, 

4)  Combinations  of  the  above. 

The  first  major  source  of  uncertainty  is 
with  respect  to  scenario.  The  time,  place 
and  opponents  in  the  next  war  are  unknow¬ 
able.  Therefore,  long-range  planning  in 
particular  must  address  a  broad  set  of  cli¬ 
mates,  geographies,  threat  capabilities  and 
US  starting  conditions.  Moreover,  there 
are  significant  uncertainties  within  existing 
planning  scenarios.  Factors  such  as  an 
opponent’s  objectives,  force  strength  and 
weapon  system  capabilities  become 
increasingly  uncertain  the  further  into  the 
future  one  projects. 

The  second  source  of  uncertainty  is 
with  respect  to  an  opponent’s  behavior. 
War  is  a  multisided,  adversarial  activity. 
Opponents  learn:  from  observation  of  past 
US  military  activity,  from  actual  combat 
and  from  US  doctrinal  pubhcations.  As  an 
opponent  learns  it  will  develop  tactics  and 
operational  concepts  that  counter  US  tech¬ 
nology  and  doctrine.  The  implication  is 
that  “scripting”  an  opponent’s  behavior  to 
high  levels  of  fidelity  using  postulated  doc¬ 
trinal  templates  could  lead  to  misinformed 
decisions,  particularly  in  campaign  and 
mission-level  models.  A  better  strategy 


would  be  to  evaluate  force  performance 
with  respect  to  multiple,  potentially  new 
enemy  doctrines. 

Technical  risk  is  also  a  significant 
source  of  uncertainty.  The  complex,  tech¬ 
nologically  intensive  nature  of  the  Joint 
Vision  2010  force  amplifies  the  signifi¬ 
cance  of  this  risk.  Two  possibilities  exist 
that  call  for  additional  comment.  The  first 
is  that  a  technology  or  system  may  not  per¬ 
form  as  expected.  As  new  technology  and 
systems  are  introduced,  analysis  must 
explore  the  robustness  of  the  entire 
warfighting  system  if  the  new  technology 
fails.  Ideally,  the  performance  of  a  force 
should  degrade  gracefully  as  technology 
failures  occur  on  the  battlefield.  Another 
possibility  is  that  combinations  of  systems 
and  technologies  might  interact  in  a  man¬ 
ner  that  has  unintended  consequences. 

Finally,  a  situation  could  arise  entailing 
combinations  of  the  three  sources  of  uncer¬ 
tainty  identified  above.  An  effective  coun¬ 
termeasure,  introduced  by  an  adversary 
against  an  emerging  technology,  is  an 
example  of  a  case  of  combinations  of 
sources.  Discussions  of  these  combina¬ 
tions  are  inappropriate  for  the  open  litera¬ 
ture. 

Conclusions 

New  technology,  weapon  systems  and 
operational  doctrine  will  continue  to 
emerge  as  we  move  into  the  21st  century. 
It  is  already  apparent  that  these  advances 
are  significantly  changing  America’s 
“warfighting  system.”  The  next-generation 
system  that  is  under  development  is  com¬ 
plex  and  highly  dependent  on  advanced 
technology.  The  new  and  emerging  “mea¬ 
sures  of  effectiveness”  that  are  descried  as 
means  objectives  are  helpful  in  increasing 
our  understanding  of  how  the  “warfighting 
system”  works.  However,  it  is  incumbent 
on  the  military  operations  research  com¬ 
munity  to  ensure  that  analyses  communi¬ 
cate  clearly  the  benefits  and  risks  of  the 
evolving  system  in  relation  to  the  nation’s 
fundamental  defense  objectives. 

Properly  structured  analyses  should  be 
presented  so  leaders  can  clearly  differenti¬ 
ate  between  means,  means  objectives  and 
fundamental  objectives  in  the  context  of 
the  decision  situation.  Additionally,  analy¬ 
sis  must  explore  uncertainties  with  respect 
to  means  objectives,  as  these  can  identify 
systemic  weaknesses  and  assist  in  their 
correction.  This  understanding  is  an 


important  contribution  to  fielding  the  best 
force  possible  for  the  future.  However,  it  is 
fundamental  objectives  that  are  reflective 
of  the  true  reasons  leaders  and  U.S.  citizens 
are  concerned  about  military  effectiveness. 
These  measures  should  be  the  foundation 
for  defense  analyses.  They  are  truly  the 
measures  of  effectiveness  that  will  be  used 
to  judge  the  success  or  failure  if,  and  when 
the  evidence  becomes  unequivicable. 
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New  Tape  on  Operations 
Research  Features  Disney, 
Sunken  Treasure  Ship 


The  Institute  for  Operations 
Research  and  the  Management  Sci¬ 
ences  (INFORMS  )  has  just  released  a 
new  tape  explaining  its  members’  pro¬ 
fession  that  includes  examples  from 
Disneyland,  the  sunken  treasure  ship 
The  Central  America  and  the  Bosnian 
conflict. 

The  tape  is  expected  to  air  on 
public  television  in  1999 
and  is  currently 
available  for  pur¬ 
chase. 

“Operations 
Research:  The  Sci¬ 
ence  and  Technolo¬ 
gy  for  Informed 
Decision  Making” 
provides  an  introduc¬ 
tion  to  the  field 
through  interviews  with 
experts  and  engaging  examples  from 
real  life. 

A  segment  on  queuing  theory,  for 
example,  includes  footage  from  Dis¬ 
neyland,  which  is  a  leader  in  applying 
operations  research  to  easing  waiting 


lines. 

Search  theory  is  illustrated  by  the 
techniques  that  METRON,  Inc.  used 
to  help  locate  The  Central  America,  a 
ship  that  sank  with  a  load  of  gold  dur¬ 
ing  a  storm  before  the  Civil 
War. 

A  relatively  uninva- 
sive  method  used  in 
early  detection  of 
breast  cancer  is 
featured  in  a 
segment  out  of 
the  University  of 
Wisconsin. 

A  US  Air  Force 
operations  research  study 
that  Bosnia  peace  negotiators 
used  to  keep  antagonists  at  the  nego¬ 
tiating  table  is  explored  as  well. 

The  tape  is  entitled  “Operations 
Research:  The  Science  and  Technolo¬ 
gy  for  Informed  Decision  Making,” 
and  can  be  ordered  from  INFORMS 
by  calling  1-800-446-3676,  faxing  1- 
410-684-2963  or  e-mailing  informs® 
informs.org. 


A  New  Name — A  New  Home — A  New  Date 

by  E,  B,  Vandiver  III,  FS 

Director,  Center  for  Army  Analysis 

In  the  December  1998  issue  of  the  PHALANX  the  Center  for  Army  Analysis 
(CAA)  formerly  the  US  Army  Concepts  Analysis  Agency  announced  its  name 
change  as  of  1  October  1998;  the  Center’s  move  to  Fort  Belvoir  on  25  March 
1999;  and  the  dedication  of  its  new  home,  Wilbur  B.  Payne  Hall,  named  in  honor 
of  Dr.  Wilbur  B.  Payne,  first  Deputy  Under  Secretary  of  the  Army  (Operations 
Research)  and  first  Director  of  the  TRADOC  Systems  Analysis  Agency 
(TRASANA).  Since  the  publication  date  of  the  bulletin  the  dedication  ceremony 
has  been  changed.  The  ceremony  will  now  take  place  on  Friday,  28  May  1999 
with  the  Payne  family  in  attendance.  Invitations  will  be  mailed  in  the  March- 
April  timeframe. 
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Operational  Field  Assessments 

John  Kreis,  IDA;  Edward  Smith,  Jr.;  LTCOL  Robert  Newnam  USAF 
e-mail:  jkreis@ida.org,  esmith@ida.org 


John  F.  Kreis,  Edward  F.  Smith, 

Institute  for  Jr.,  Institute  for 

Defense  Analyses  Defense  Analyses 


Robert  M.  Newnam,  Lt.  Col., 

USAF,  Office  of  the  Director, 
Operational  Test  and  Evaluation 

Operational  Field  Assessments 
(OFAs)  began  in  1997,  after  Con¬ 
gress  authorized  the  DoD’s  Direc¬ 
tor  of  Operational  Test  and  Evaluation 
(DOT&E)  to  spend  up  to  $3  million  to 
demonstrate  the  program.  Several  OFAs 
have  been  completed  with  substantial 
results,  but  OFAs  are  not  widely  known  in 
the  test  and  evaluation  community.  Our 
purpose  here  is  to  explore  how  these 
assessments  relate  to  operational  test  and 
evaluation. 

An  Operational  Field  Assessment  is  not 
a  test,  and  OFAs  will  not  replace  opera¬ 
tional  test  and  evaluation.  Rather,  OFAs 
are  coordinated  efforts  between  a  group  of 
partners  in  DoD  to  support  and  assess 
operational  requirements  put  forward  by 
the  CINCs.  An  OFA  can  be  a  field  assess¬ 
ment  or  an  experimental  application  of 
joint  operational  concepts  to  weigh  their 
effectiveness  against  certain  threats.  An 
OFA  may  also  demonstrate  hardware  or 
software  applications  or  be  used  to  prove 
operational  or  doctrinal  concepts  within  a 
CINC’s  area  of  interest.  Above  all,  OFAs 
are  very  limited  in  scope,  their  purpose 
usually  being  to  provide  quick  support  to 
the  CINCs  on  near-term  operational  issues 
that  otherwise  might  take  years  to  achieve 
using  joint  exercises  or  joint  testing. 

Background 

During  1996,  one  of  DOT&E’s  princi¬ 
pal  initiatives  was  to  develop  methods  to 


help  the  operational  test  and  evaluation 
(OT&E)  community  more  directly  support 
the  theater  or  functional  commanders-in- 
chief  and  their  immediate  subordinate 
commands.  The  OFA  was  one  of  the  prin¬ 
cipal  concepts  chosen  to  accomplish  this 
objective.  The  basic  tenet  of  the  OFA  pro¬ 
gram  is  to  support  the  CINCs  in  assessing 
how  best  to  employ  military  assets  in  cer¬ 
tain  environments  or  against  particular 
threats.  OFAs  also  are  designed  to  address 
adaptations  of  existing  systems  for  a  given 
mission  to  better  understand  the  capabili¬ 
ties  of  those  systems  in  new  or  unusual  sit¬ 
uations.  They  also  address  employment 
options  that  increase  the  capabilities  of  US 
military  equipment.  Generally,  formal 
operational  testing  and  evaluation  of  major 
weapon  systems  does  not  anticipate  the 
needs  addressed  by  OFAs  because  not 
every  operational  scenario  can  be  tested, 
and  new  operational  requirements  continu¬ 
ally  arise  as  a  result  of  the  world’s  chang¬ 
ing  political  and  military  factors. 

Defining  OFAs 

The  OFA  partnership  operates  on  a 
memorandum  of  understanding  between 
the  DOT&E,  the  Director,  Defense  Intelli¬ 
gence  Agency  (DIA),  the  Director,  Nation¬ 
al  Security  Agency  (NSA)  and  the  Direc¬ 
tor,  National  Reconnaissance  Office 
(NRO).  The  Joint  Chiefs  of  Staff  (JCS)  are 
OFA  participants  as  well.  At  the  pro¬ 
gram’s  outset,  the  JCS’s  Vice  Chairman 
designated  the  J-8  as  the  JCS’s  point  of 
contact  for  OFAs.  The  J-8  and  the  OFA 
partners  then  developed  a  coordination 
process  to  address  the  CINCs’  requests  for 
operational  assessments.  With  the  Unified 
Commands,  several  initial  OFA  require¬ 
ments  were  identified,  and  the  partners 
pursued  the  first  of  these  assessments  in 
early  1997. 

The  OFA  concept  not  only  has  the 
potential  to  shorten  the  time  needed  to  ana¬ 
lyze  the  utility  of  a  given  technology  or 
process,  it  also  may  provide  data  or  help  in 
forming  concepts  of  employment  that  can 
measurably  improve  the  preparation  of  for¬ 
mal  operational  test  and  evaluation,  while 
reducing  the  cost  of  such  testing.  Since  the 


concepts  or  systems  being  assessed  during 
an  OFA  may  well  be  used  operationally  in 
any  number  of  critical  situations,  using 
proven  operational  evaluation  methods  as 
the  background  for  assessing  the  potential 
utility  of  newly  developing  doctrines,  prac¬ 
tices  or  systems  is  ideal.  Because  there  are 
many  similarities  between  OFAs  and  oper¬ 
ational  testing,  the  DOT&E  became  the 
executive  agent  and  DoD  sponsor  for  the 
OFA  partners. 

Generally,  a  CINC  has  few  alternatives 
to  an  OFA  if  he  needs  quick  answers  to 
joint  operational  problems.  Neither  large, 
joint  command  exercises  nor  joint  test  and 
evaluation  can  meet  the  CINCs’  time-criti¬ 
cal  needs,  since  tests  and  exercises  are  too 
wide  ranging  and  often  need  years  of  plan¬ 
ning  preparation.  A  joint  test  and  evalua¬ 
tion,  for  example,  normally  runs  for  three 
years,  and  both  tests  and  exercises  are 
closely  controlled  toward  specific  goals. 
An  exercise  normally  has  complex  training 
objectives,  and  will  not  provide  the  flexi¬ 
bility  for  experimentation  or  the  employ¬ 
ment  of  new  technology  or  techniques. 
Tests  are  usually  aimed  at  specific  evalua¬ 
tions,  and  may  not  lend  themselves  to  the 
add-on  of  assessing  an  OFA  concept  or 
technology. 

OFAs,  on  the  other  hand,  are  small, 
flexible  and  fast  —  they  are  organized  to 
meet  a  warfighter’s  needs  now.  Addition¬ 
ally,  OFAs,  if  planned  adequately,  can 
demonstrate  some  aspects  of  a  new  tech¬ 
nology’s  or  concept’s  operational  effec¬ 
tiveness.  Thus,  OFAs  can  provide  advance 
data  and  understanding  of  candidate  sys¬ 
tems  should  they  be  developed  further  and 
brought  to  fiiU  operational  test  and  evalua¬ 
tion,  as  some  will.  Already,  several  OFAs 
have  been  completed,  producing  a  number 
of  worthwhile  observations. 

Practical  Results 

The  first  OFA,  based  on  a  request  fi:om 
the  Commander-in-Chief,  US  European 
Command,  focused  on  unusual,  hybrid, 
surface-to-air  threats  that  aircraft  of  the 
European  Command  and  the  US  Central 
Command  might  face  in  their  theaters  of 
operation.  This  effort  studied  radars  and 
weapons  not  normally  used  together  as  a 
system,  but  assembled  via  a  supporting 
network  to  disguise  their  presence  and  pre¬ 
sent  an  unexpected  threat  to  US  or  Allied 
aircraft.  The  basic  idea  for  this  potential 

(See  ASSESSMENTS,  p.  32) 
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enemy  tactic  was  not  new,  similar  arrange¬ 
ments  having  been  used  by  the  North  Viet¬ 
namese  against  American  aircraft,  and  then 
by  the  Egyptians  against  the  Israeli  Air 
Force  during  the  1973  Middle  East  War. 
This  assessment,  however,  centered  on 
newer  radar  and  weapon  combinations  that 
had  not  been  seen  in  combat  and  whose 
dangers  and  operating  characteristics  were 
suspected  but  unverified.  Since  these  were 
extraordinary  weapon  arrangements,  the 
usual  intelligence  information  was  not 
complete  enough  to  judge  the  threat  they 
presented;  past  experience  meant  that  a 
closer  look  in  a  field  trial  was  warranted. 
Within  seven  weeks  of  the  identification  of 
support  funding,  a  five-day  experiment  at 
Fort  Bliss  collected  data  on  three  unusual 
combinations  of  radars  and  weapons. 
Immediate  results  included  both  recom¬ 
mended  improvements  in  aircrew  operat¬ 
ing  procedures  and  a  package  of  warning 
instrumentation  for  aircraft,  as  well  as  a 
refreshed  understanding  by  the  OFA  part¬ 
ners  of  the  warfighters’  intelligence  needs. 

Another  OFA,  and  one  in  which  the 
authors  participated,  set  out  to  demonstrate 
the  usefulness  of  a  computer-based  service 
that  would  provide  a  distributed,  collabora¬ 
tive  planning  (DCP)  capability.  This 
demonstration  was  actually  of  a  prototype 
system  being  considered  for  a  potential 
acquisition  program.  A  unified  command 
wanted  this  system  to  reduce  plan  prepara¬ 
tion  time  and  to  increase  the  time  available 
for  course  of  action  assessments  and  mis¬ 
sion  rehearsals.  As  with  the  hybrid  sur¬ 
face-to-air  threat,  this  assessment,  too,  was 
organized  on  short  notice  —  less  than  three 
months.  Again,  as  with  the  assessment  of 
surface-to-air  threats,  this  request  came 
from  a  CINC.  The  assessment’s  setting, 
however,  differed  greatly  in  that  it  was  part 
of  a  major  joint  force  exercise  in  the  south¬ 
western  United  States  rather  than  a  sepa¬ 
rate,  controlled  activity. 

Unfortunately,  the  exercise  was  not  well 
suited  to  introducing  the  prototype  system; 
the  personnel  using  it  lacked  familiarity 
with  it  and  tactics,  techniques  and  proce¬ 
dures  for  its  use  had  not  been  developed. 
Moreover,  the  candidate  system  itself 
lacked  the  maturity  to  withstand  the  stress¬ 
es  it  faced  in  an  operational  versus  a 
demonstration  venue.  The  upshot  was  that 
the  DCP  attempt  could  not  keep  up  with 


the  operating  tempo.  The  problem  high¬ 
lighted  a  clear  consideration  for  future 
OFAs:  the  immediate  difficulty  with 
attaching  a  new  methodology,  system,  or 
practice  to  an  exercise  is  that  if  it  fails  to 
meet  the  players’  expectations  and  needs, 
they  will  abandon  it  as  they  seek  to  fulfill 
their  mission  objectives  in  the  old  and  well 
understood  ways.  That  is  what  happened 
in  this  case.  Although  the  system  being 
assessed  failed,  the  OFA  succeeded.  It 
gave  the  CINC  information  on  which  to 
base  decisions  about  how  to  change  and 
pursue  future  development  and  acquisition 
of  a  distributed,  collaborative  planning 
capability  that  his  command  needs. 

Other  OFAs  already  approved  from  the 
first  year’s  authorizations  are  to  be  con¬ 
ducted  in  the  near  future.  One,  requested 
by  the  US  Southern  Command,  is  to  pro¬ 
vide  an  independent  assessment  of  that 
command’s  exercise  planning  and  execu¬ 
tion  processes,  and  to  suggest  areas  where 
those  processes  may  be  improved  through 
the  use  of  techniques  often  employed  by 
the  operational  test  and  evaluation  commu¬ 
nity.  This  assessment  will  comprise  three 
different  activities  in  1998:  a  peacekeep¬ 
ing  exercise  in  Central  America,  a  humani¬ 
tarian  and  disaster  relief  exercise  in  the 
Caribbean,  and  an  engineer  exercise  in 
Central  America.  Reviews  will  be  pre¬ 
pared  for  the  planning,  execution,  and  post¬ 
execution  phases  of  each  with  lessons 
learned,  all  aimed  at  improving  the  process 
for  multi-participant,  multi-national  efforts. 

An  additional  OFA  will  demonstrate  the 
potential  of  a  low-cost,  airborne  podded 
radar  to  conduct  air-to-air  intercepts. 
Another  is  studying  the  ability  of  the  relo¬ 
catable  over-the-horizon  radar  to  detect 
and  track  targets  at  extended  ranges  of 
2,500  nautical  miles.  So  far,  this  radar  has 
demonstrated  the  ability  to  monitor  the  aer¬ 
ial  activities  of  suspected  drug  smugglers 
in  order  to  warn  interdiction  agencies  of 
suspicious  movements. 

Lessons  Learned 

Operational  Field  Assessments  can 
range  from  a  technical  review  to  answer 
limited  questions  about  certain  hardware  or 
system  processes  (rather  narrow,  in  other 
words),  to  an  assessment  that  seeks  to  sur¬ 
vey  wider  avenues,  including  changes  to 
hardware,  software,  or  doctrinal  and  opera¬ 
tional  concepts.  These  broad  assessments 
demand  awareness  of  a  number  of  prob¬ 


lems  that  can  arise  unexpectedly  when 
bringing  a  new  concept  through  its  initial 
scrutiny  to  operational  trials.  The  effects 
of  transitions  from  developmental  or 
experimental  use  to  operational  employ¬ 
ment  should  not  be  minimized  or  wished 
away  just  because  the  candidate  seems 
right  for  the  task.  In  fact,  such  transitions 
can  be  complex.  Not  anticipating  them  can 
cause  the  assessment  to  collapse  —  as  hap¬ 
pened  with  the  DCP  system  —  and  an 
otherwise  promising  idea  to  fail.  From 
what  we  have  seen  recently,  the  transitions 
an  OFA  candidate  experiences  as  it  is 
brought  from  an  idea  into  the  world  of 
physical  experiments  parallels  the  types  of 
transitional  difficulties  seen  when  moving 
a  new  system  from  developmental  testing 
into  OT&E. 

OFAs,  like  operational  tests,  cannot  be 
entered  into  haphazardly.  They  need  a 
structure,  and  that  structure  roughly  resem¬ 
bles  that  of  a  plan  for  an  operational  test. 
OFA  preparation  needs  to  set  forth  require¬ 
ments,  resources,  evaluation  criteria  and  a 
methodology.  Moreover,  the  assessment 
must  be  internally  disciplined  and  built 
according  to  an  agreed-upon  concept  of 
operation.  The  internal  discipline  is  need¬ 
ed  so  that  the  potential  users  do  not  deceive 
themselves,  or  are  not  deceived,  into  think¬ 
ing  that  the  system  has  utility  based  upon 
unsupported  claims. 

A  critical  issue  to  be  settled  at  the  outset 
of  OFA  preparation  is  for  the  potential  user 
to  articulate  the  results  he  expects  to  see 
demonstrated.  If  this  can  be  done  clearly, 
the  requirements  can  be  converted  into 
testable  or  observable  issues.  Evaluators 
can  then  develop  suitable  pass/fail  criteria 
to  measure  performance  during  the  assess¬ 
ment. 

Understanding  the  requirements  and 
expected  levels  of  performance  will  allow 
construction  of  the  OFA’s  framework. 
With  the  distributed,  collaborative  plan¬ 
ning  system,  for  example,  the  authors 
reviewed  the  available  material  about  the 
candidate  system.  Next,  we  drafted  a  set  of 
operational  effectiveness  and  operational 
suitability  criteria,  some  of  which  we 
thought  would  be  addressed  during  the 
immediately  upcoming  demonstration.  We 
designed  these  criteria  to  be  used  as  well 
for  guiding  assessments  of  the  advanced 
system,  should  the  command  create  a  full- 
fledged  development  project.  In  this  way, 
we  hoped  that  the  results  could  be  matched 
against  the  command’s  overall  require- 
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ments  and  evaluation  issues  so  as  to  judge 
the  true  state  of  the  candidate  and  discern 
its  maturity.  Information  from  the  OFA 
also  was  intended  to  help  create  a  baseline 
for  future  development  and  allow  the 
command  to  adjust  its  expectations  for  the 
long-term  project. 

One  of  the  more  striking  aspects  of  the 
distributed,  collaborative  planning  OFA 
was  the  marked  lack  of  coordination  and 
organizational  discipline  leading  up  to  the 
exercise.  This  was  a  stark  contrast  to  the 
normal,  step-by-step  process  that  turns  on 
a  major  operational  test  plan.  Much  of  the 
OFA’s  disorganization  stemmed  from  the 
command’s  hasty  decision  to  try  the  sys¬ 
tem  based,  in  good  measure,  on  the  con¬ 
tractor’ s  assurances  that  it  was  fully  devel¬ 
oped  and  ready  for  operational  use;  in 
truth,  it  had  a  way  to  go  in  this  regard. 
Additionally  the  exercise  had  been  so 
quickly  arranged  that  most  of  the  players 
did  not  know  how  to  use  the  new  system. 

Further  adding  to  the  problems  was 
that,  lacking  an  approved  ORD,  the 
requirements  to  be  judged  during  the 
assessment  were  sometimes  vague  or 
uncertain;  at  other  times,  the  requirements 
were  based  on  the  opinions  of  some  of  the 
players,  yet  not  clearly  defined  in  the 
minds  of  others.  This  pattern  probably 
will  not  be  unusual.  Many  OFA  candi¬ 
dates  are,  by  their  very  nature,  ideas  or 
semi-developed  software  or  hardware  for 
which  no  system  ORD  would  be  possible 
or  appropriate.  This  can  make  under¬ 
standing  performance  a  challenge,  and  is 
all  the  more  reason  to  work  closely  with 
the  prospective  user  to  develop  a  good 
plan  that  sets  forth  requirements, 
resources,  evaluator  criteria,  and  an  appro¬ 
priate  assessment  methodology.  Just 
because  OFAs  are  less  formal  than  other 


test  and  evaluation  settings,  does  not  neces¬ 
sarily  mean  that  all  of  them  can  be  pre¬ 
pared  on  short  notice.  Some  assessments 
simply  require  more  planning  time;  the 
complexity  of  each  needs  to  be  carefully 
judged.  Figure  1  illustrates  a  preparation 
sequence  that  would  have  been  more  rea¬ 
sonable  for  the  DCP  system,  for  example. 

When  we  summed  up  the  outcome  of 
the  DCP  assessment  and  reviewed  other 
such  projects,  it  became  apparent  that  a 
number  of  entrance  criteria  steps  are  need¬ 
ed  if  an  OFA  is  to  be  successful: 

•  Preparing  usable  concepts  of  operation 
and  employment,  plus  implementing 
procedures  (even  if  these  are  no  more 
than  the  partially  developed  ideas  of  a 
CINC  or  his  staff).  Lacking  an  ORD  or 
CONOPS  makes  it  all  the  more  neces- 
saiy  to  prepare  some  clear  objectives  for 
the  candidates. 

•  Devising  some  method  of  certification  to 
indicate  that  the  candidate  is  ready  for 
assessment,  much  like  an  operational  test 
readiness  review,  although  not  nearly  as 
detailed. 

•  Implementing  training  programs  and 
packages  to  ensure  that  players  know 
how  to  use  the  system. 

•  Preparing  adequate  databases  that  may 
be  needed,  such  as  map  and  terrain  infor¬ 
mation,  and  the  availability  of  data 
sources  the  users  might  need  during  the 
assessment. 

•  Applying  effective  system  management 
procedures  and  capabilities. 

•  Providing  adequate  technical  support  for 
the  systems  and  players. 

•  Providing  for  recovery  and  backup  in  the 
event  of  system  crash. 


•  Employing  an  agreed-to  doctrine  that 
can  be  used  to  tie  the  many  tasks  into  a 
cohesive  whole,  able  to  carry  out  the 
integrated  mission. 

A  candidate  system’s  proponents  must 
not  overlook  the  fact  that  when  a  new  idea, 
process,  or  technique  is  adopted  quickly,  it 
often  is  not  a  part  of  the  command’s  daily 
operation.  The  command’s  leaders  often 
lack  the  time  and  understanding  to  incorpo¬ 
rate  it  into  their  routine,  and  have  not 
adapted  their  thinking  to  include  it.  As  an 
addition  to,  rather  than  a  part  of,  the  unit’s 
normal  operation,  a  new  system  may  well 
not  receive  the  full  consideration  it  needs 
to  be  successful  in  a  complex  exercise  set¬ 
ting.  To  succeed  in  such  an  endeavor, 
sponsors  and  evaluators  alike  need  to 
understand  the  importance  of  getting  the 
system  or  process  to  be  assessed  into  early 
use,  so  that  people  understand  it  and  are 
willing  to  integrate  it  into  their  normal  rou¬ 
tine  ( to  trust  it  in  the  face  of  risk). 

Military  innovation  via  technological 
change  holds  both  promise  and  serious 
potential  problems.  Evaluators,  if  they  are 
to  influence  the  promise  or  understand  the 
risks  and  problems,  need  to  become 
involved  in  the  assessment  as  early  in  its 
planning  as  possible.  Even  so,  early 
involvement  by  itself  will  not  be  enough. 
The  assessment  has  to  be  made  with  full 
agreement  among  all  participants  of  the 
goals  and  objectives,  and  the  candidate’s 
limitations  must  be  well  understood. 

Getting  back  to  the  earlier  distinction 
between  narrowly  defined  OFAs  intended 
to  evaluate  something  like  a  potential 
threat,  as  opposed  to  broader  OFAs  that 
will  lead  to  changes  of  operational  doc¬ 
trine,  we  must  remember  the  most  impor- 

(See  ASSESSMENTS,/?.  35) 
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Figure  1.  For  some  OFAs,  preparations  can  be  deceptively  complex.  Attempting  to  expedite  the  distributed,  collaborative  planning 
system’s  operational  field  assessment,  the  sponsor  provided  only  two  month’s  preparation;  the  nine  months  shown  in  the  timeline 
above  would  have  been  more  realistic. 
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tant  issue:  if  the  candidate  in  the  OFA  is  to 
be  used  by  troops  who  will  depend  upon  it 
for  mission  success,  even  for  their  lives,  it 
must  prove  itself  to  be  clearly  advanta¬ 
geous  in  a  rigorous  operational  setting. 
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THE  LAST  WORD 


Julian  Palmore 

Since  its  inception  the  John  K. 
Walker  Award  has  heightened  our 
need  to  have  technical  articles  of 
the  highest  quality  in  PHALANX.  This  is 
tough  to  do.  It's  tough  for  authors  because 
PHALANX  only  has  space  for  around 
3000  words  per  article  which  translates 


into  around  3-4  pages.  In  a  36  page  issue 
we  can  print  only  4-5  technical  articles. 
It's  tough  also  because  the  articles  must  be 
timely,  important  and  interesting  to  the 
PHALANX  audience.  PHALANX  is  a  Bul¬ 
letin.  There  are  other  things  than  technical 
articles  that  are  timely,  important  and 


interesting.  So  we  maintain  a  balance. 
PHALANX  Online  will  help  take  the  over¬ 
load  thanks  to  CAPT  Lee  Dick.  I  encour¬ 
age  all  of  you  who  want  to  write  for  PHA¬ 
LANX  to  do  so.  We  need  your  support!  O 
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